Image processing apparatus, image processing method, and program for generating multi-view point image

ABSTRACT

There is provided an image processing apparatus including a left eye image input unit configured to input a left eye image (L image) which is a left eye image signal applied to three-dimensional image display, a right eye image input unit configured to input a right eye image (R image) which is a right eye image signal applied to three-dimensional image display, a parallax information generating unit configured to generate parallax information from the left eye image (L image) and the right eye image (R image), and a virtual view point image generating unit configured to receive the left eye image (L image), the right eye image (R image), and the parallax information, and generate virtual view point images including a view point image other than view points of the received LR images.

BACKGROUND

The present disclosure relates to an image processing apparatus, animage processing method, and a program, and more particularly, to animage processing apparatus, an image processing method, and a programfor generating a multi-view point image which is applied tothree-dimensional (3D) image display.

A naked eye-type 3D display apparatus has been put into practice. Thenaked eye-type 3D display apparatus allows a user to perceive astereoscopic image without wearing glasses in three-dimensional (3D)image display processing. The naked eye 3D display apparatus includes,for example, a lenticular sheet or a parallax barrier (parallax barrier)on a display surface, which controls images entering into the left eyeand the right eye in accordance with a viewing/listening position.

With such method, however, a correct stereoscopic vision can be obtainedonly at a limited viewing/listening position with respect to a display.Therefore, when a user's observation position is located at a positiondifferent from a specified position, reversed vision and crosstalkoccurs. In the reversed vision, a right eye image enters into the lefteye, and a left eye image enters into the right eye. In the crosstalk, aleft eye image and a right eye image are mixed.

In order to solve this problem, a configuration has been suggested togenerate and display not only a standard left eye image and a standardright eye image corresponding to a regular observation position but alsoan image from a new view point which is configured not to produce anycrosstalk when observed from other observation positions.

Not only an original set of a left eye image and a right eye image butalso images of other virtual view points are generated as multi-viewpoint images, and an appropriate set of a left eye image and a right eyeimage according to a user's observation position is selectable fromthese multi-view point images, in accordance with the observationposition, whereby images are displayed while reducing the reversedvision and the crosstalk.

In other words, this allows a user to observe a different pair of a lefteye image and a right eye image in accordance with the user'sobservation position, so that even when the user's observation positionis changed, this allows the left eye and the right eye of the observerto observe a left eye image and a right eye image according to eachobservation position.

More specifically, based on the original images for two view pointswhich are input to a display apparatus or an image processing apparatus,i.e., two view point images including a left eye image (L image) and aright eye image (R image) for 3D image display, view point images forvirtual view points are generated in addition to these two view points.For example, multi-view point images for ten different view pointsincluding the original LR images are generated.

A user observes a combination of appropriate two images among thegenerated multi-view point images in accordance with a user'sobservation position with respect to the display, whereby 3D images canbe displayed and can be observed while reducing crosstalk in which aleft eye image and a right eye image are mixed, at various observationpositions.

For example, Japanese Patent Application Laid-Open No. 2006-115198discloses a method for inputting an original left eye image (L image)and an original right eye image (R image), executing parallax detectionfrom these two images, and generating images for multiple virtual viewpoints, on the basis of the detected parallax information. Morespecifically, parallax is detected from the two received original 3Dimages including the left eye image (L image) and the right eye image (Rimage), and determining virtual view point positions different from thereceived LR images, on the basis of the amount of crosstalk and thefusional parallax range.

In the processing described in this Japanese Patent ApplicationLaid-Open No. 2006-115198, however, the quality of the generated virtualview point images is not taken into consideration, and the processingdescribed in this Japanese Patent Application Laid-Open No. 2006-115198is configured to determine the virtual view point positions using thecenter of the left eye image and the right eye image as a reference.Therefore, the quality of the generated virtual view point images isreduced, and an image which can be hardly observed may be displayed.

There is a close relationship between the virtual view point positionand the image quality.

For example, where the view point position of the received L image is0.0, and the view point position of the received R image is 1.0, therelationship between an image for a newly generated virtual view pointand its image quality has the following features.

(Feature 1) At a virtual view point position between 0.0 and 1.0, i.e.,between the L image (0.0) and the R image (1.0), a virtual view pointimage of 0.5 which is the central position of the LR images has thelowest image quality as compared with other virtual view pointpositions.

(Feature 2) At a virtual view point position which is equal to or lessthan 0.0 and equal to or more than 1.0 at the left side of the L imageor at the right side of the R image, the farther the position is awayfrom the L image or the R image, the lower the quality of video becomes.

Such relationship between the virtual view point position and the imagequality results from, for example, the precision of the parallaxdetection and the amount of occlusion region included in the image.

It should be noted that when the view point position of 0.0, theoriginal received left eye image can be used as it is, and at the viewpoint position of 1.0, the original received right eye image can be usedas it is. Therefore, at these positions, the image quality becomes thehighest.

Japanese Patent Application Laid-Open No. 2006-115198 suggests a methodfor detecting the maximum amount of parallax from 3D images of anoriginal received left eye image (L image) and an original receivedright eye image (R image) and determining virtual view point positionsso that the maximum parallax is accommodated within the amount ofcrosstalk and the fusional parallax range. In other words, thisdiscloses a method for determining a view point interval of virtual viewpoint image generated according to the maximum amount of parallax of thereceived LR images.

However, when the maximum parallax is detected from the original LRimages, an image in an image region having the maximum parallax andlikelihood an image region attracts attention are not taken intoconsideration. Therefore, for example, the following problems occur.

-   -   During the maximum parallax detection, the size of area of the        image region having the maximum parallax is not taken into        consideration. Therefore, when an object having a small size of        area has the maximum parallax, the virtual view point interval        may be reduced more than necessary, in accordance with the        existence of the maximum parallax image region that hardly        affects the visual appearance.    -   In addition, the likelihood of getting attraction to the image        region having the maximum parallax is not taken into        consideration during the maximum parallax detection. Therefore,        when an image region that hardly attracts attention in terms of        visual appearance has the maximum parallax, the virtual view        point interval may be reduced or increased more than necessary,        in accordance with the maximum parallax information of the image        region that hardly affects the visual appearance.

Japanese Patent Application Laid-Open No. H9-121370 discloses a methodusing an original received left eye image (L image) and an originalreceived right eye image (R image) to maintain parallax within afusional range by moving these images in parallel (shifting).

By generating virtual view point images by means of shift processingdisclosed in Japanese Patent Application Laid-Open No. H9-121370, theoffset of the parallax distribution can be adjusted, i.e., the offsetadjustment can be done to move an too-deep image to the viewer's side asa whole. However, since the extension of the parallax distribution maynot be adjusted, the following glitches may occur: the entire image mayshift too much to the viewer's side or may move too much to the deeperside as a result of the offset adjustment.

SUMMARY

The present disclosure is to solve, for example, the above problems, andprovides an image processing apparatus, an image processing method, anda program having a configuration of performing generating processing ofmulti-view point images based on a left eye image (L image) and a righteye image (R image) for 3D image, wherein the multi-view point imagesare generated upon determining virtual view point positions in view of,for example, an image quality, an appropriate amount of parallax, or aregion of an image which is likely to attract attention.

According to a first embodiment of the present disclosure, there isprovided an image processing apparatus including a left eye image inputunit configured to input a left eye image (L image) which is a left eyeimage signal applied to three-dimensional image display, a right eyeimage input unit configured to input a right eye image (R image) whichis a right eye image signal applied to three-dimensional image display,a parallax information generating unit configured to generate parallaxinformation from the left eye image (L image) and the right eye image (Rimage), and a virtual view point image generating unit configured toreceive the left eye image (L image), the right eye image (R image), andthe parallax information, and generate virtual view point imagesincluding a view point image other than view points of the received LRimages. The virtual view point image generating unit determines virtualview point positions by means of processing in view of at least one ofimage qualities of virtual view point images, an appropriate amount ofparallax, or an image weight according to an image region, and generatesthe virtual view point images corresponding to the determined virtualview point positions.

The virtual view point image generating unit calculates an image qualityevaluation value Q indicating an image quality of a virtual view pointimage, calculates a virtual view point interval G by applying thecalculated image quality evaluation value Q, and determines the virtualview point position on the basis of the calculated virtual view pointinterval G.

The virtual view point image generating unit calculates the imagequality evaluation value Q by applying information of at least one ofreliability degree information of the parallax information or thegenerated virtual view point image information.

The virtual view point image generating unit calculates, as anappropriate amount of parallax, a smaller value of a fusional parallaxamount and a crosstalk allowable amount, calculates a virtual view pointinterval G by applying the calculated appropriate amount of parallax,and determines the virtual view point position on the basis of thecalculated virtual view point interval G.

The virtual view point image generating unit calculates, as anappropriate amount of parallax, a smaller value of a fusional parallaxamount and a crosstalk allowable amount, calculates a virtual view pointinterval G by applying the calculated appropriate amount of parallax,and determines the virtual view point position on the basis of thecalculated virtual view point interval G.

The image processing apparatus includes a weight information generatingunit configured to calculate image weight information according to animage region. The virtual view point image generating unit calculates aweighted parallax distribution obtained by correcting the parallaxinformation by applying the image weight information, calculates avirtual view point interval G by applying the appropriate amount ofparallax and a maximum value of parallax calculated from the calculatedweighted parallax distribution, and determines the virtual view pointposition on the basis of the calculated virtual view point interval G.

The weight information generating unit generates image weightinformation in which a weight in unit of image region is set accordingto a position of an image or image weight information according to asubject included in an image.

The virtual view point image generating unit determines a first virtualview point position by means of processing in view of at least one of animage quality of a virtual view point image, an appropriate amount ofparallax, or an image weight according to an image region, determines asecond virtual view point position of non-regular interval by means ofnon-linear mapping processing performed on the determined first virtualview point position, and generates a virtual view point imagecorresponding to the determined second virtual view point position ofthe non-regular interval.

The virtual view point image generating unit determines the virtual viewpoint position by means of processing in view of at least one of animage quality of the virtual view point image, an appropriate amount ofparallax, or an image weight according to an image region, calculates anamount of parallel movement on the basis of parallax distribution datacalculated from the parallax information, executes moving processing ofthe parallax distribution between the virtual view point images of therespective virtual view point positions on the basis of the calculatedamount of parallel movement, and generates virtual view point imagesreflecting a moving processing result of the parallax distribution data.

According to the second embodiment of the present disclosure, there isprovided an image-capturing apparatus including an image-capturing unitconfigured to capture a left eye image (L image) which is a left eyeimage signal and a right eye image (R image) which is a right eye imagesignal, which are applied to three-dimensional image display, a left eyeimage input unit configured to input, from the image-capturing unit, theleft eye image (L image) which is the left eye image signal applied tothe three-dimensional image display, a right eye image input unitconfigured to input, from the image-capturing unit, the right eye image(R image) which is the right eye image signal applied to thethree-dimensional image display, a parallax information generating unitconfigured to generate parallax information from the left eye image (Limage) and the right eye image (R image), and a virtual view point imagegenerating unit configured to receive the left eye image (L image), theright eye image (R image), and the parallax information, and generatevirtual view point images including a view point image other than viewpoints of the received LR images. The virtual view point imagegenerating unit determines virtual view point positions by means ofprocessing in view of at least one of image qualities of virtual viewpoint images, an appropriate amount of parallax, or an image weightaccording to an image region, and generates the virtual view pointimages corresponding to the determined virtual view point positions.

According to the third embodiment of the present disclosure, there isprovided an image processing method with which an image processingapparatus generates multi-view point images, the image processing methodincluding inputting, by a left eye image input unit, a left eye image (Limage) which is a left eye image signal applied to three-dimensionalimage display, inputting, by a right eye image input unit, a right eyeimage (R image) which is a right eye image signal applied tothree-dimensional image display, generating, by a parallax informationgenerating unit, parallax information from the left eye image (L image)and the right eye image (R image), and receiving, by a virtual viewpoint image generating unit, the left eye image (L image), the right eyeimage (R image), and the parallax information, and generating virtualview point images including a view point image other than view points ofthe received LR images. In the virtual view point image generating step,virtual view point positions are determined by means of processing inview of at least one of image qualities of virtual view point images, anappropriate amount of parallax, or an image weight according to an imageregion, and the virtual view point images corresponding to thedetermined virtual view point positions are generated.

According to the fourth embodiment of the present disclosure, there isprovided a program for causing an image processing apparatus to generatemulti-view point images, the program including causing a left eye imageinput unit to input a left eye image (L image) which is a left eye imagesignal applied to three-dimensional image display, causing a right eyeimage input unit to input a right eye image (R image) which is a righteye image signal applied to three-dimensional image display, causing aparallax information generating unit to generate parallax informationfrom the left eye image (L image) and the right eye image (R image); andcausing a virtual view point image generating unit to receive the lefteye image (L image), the right eye image (R image), and the parallaxinformation, and generate virtual view point images including a viewpoint image other than view points of the received LR images. In thevirtual view point image generating step, virtual view point positionsare determined by means of processing in view of at least one of imagequalities of virtual view point images, an appropriate amount ofparallax, or an image weight according to an image region, and thevirtual view point images corresponding to the determined virtual viewpoint positions are generated.

It should be noted that the program according to an embodiment of thepresent disclosure is, for example, a program that can be provided by astorage medium or a communication medium provided in a computer-readableformat to a general purpose system that can execute various programcodes. By providing such programs in the computer-readable format,processing according to the programs is achieved on a computer system.

Other objects, features, and advantages of the present disclosure willbecome apparent from more detailed description based on attacheddrawings and embodiments of the present disclosure explained below. Inthis specification, a system is a logical configuration of a set ofmultiple apparatuses, and an apparatus of each configuration is notnecessarily limited to be provided within the same housing.

According to a configuration of an embodiment of the present disclosure,a configuration for generating multi-view point images based on LRimages of three-dimensional images is achieved.

More specifically, for example, a virtual view point image generatingunit is provided, wherein the virtual view point image generating unitreceives a left eye image (L image) and a right eye image (R image)which are applied to three-dimensional image display, generates parallaxinformation on the basis of the left eye image (L image) and the righteye image (R image), and uses the LR image and the parallax informationto generate virtual view point images including view point images otherthan the view points of the received LR images. The virtual view pointimage generating unit determines the virtual view point positions bymeans of processing in view of at least one of an image quality of avirtual view point image, an appropriate amount of parallax determinedin view of a fusional parallax amount and a crosstalk allowable amount,and an image weight according to an image region of a subject and thelike and a position of an image, and generates virtual view point imagescorresponding to the determined virtual view point positions.

With such processing, optimum virtual view point images according torespective observation positions, i.e., high-quality virtual view pointimages of comfortable parallax ranges, can be generated.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a figure illustrating a flowchart explaining a processingsequence executed by an image processing apparatus;

FIG. 2 is a figure explaining an example of generating processing ofvirtual view point images;

FIG. 3 is a figure explaining an example of display processing ofmulti-view point images;

FIG. 4 is a figure explaining an example of display processing ofmulti-view point images;

FIG. 5 is a figure illustrating a flowchart for explaining an example ofprocessing of virtual view point position determining processingsequence executed by an image processing apparatus;

FIG. 6 is a figure illustrating an image quality of a virtual view pointimage;

FIG. 7 are figures explaining an example of calculation processing of animage quality of a virtual view point image;

FIG. 8 is a figure explaining an example of calculation processing of animage quality of a virtual view point image;

FIG. 9 is a figure explaining an example of determining processing ofvirtual view point positions corresponding to respective view pointpositions of virtual view point images of N view points;

FIG. 10 is a figure explaining an example of determining processing ofvirtual view point positions corresponding to respective view pointpositions of virtual view point images of N view points;

FIG. 11 is a figure explaining an example of determining processing ofvirtual view point positions corresponding to respective view pointpositions of virtual view point images of N view points;

FIG. 12 is a figure explaining an example of determining processing ofvirtual view point positions corresponding to respective view pointpositions of virtual view point images of N view points;

FIG. 13 is a figure explaining an example of determining processing ofvirtual view point positions corresponding to respective view pointpositions of virtual view point images of N view points;

FIG. 14 is a figure illustrating a flowchart for explaining an exampleof processing of virtual view point position determining processingsequence executed by an image processing apparatus;

FIG. 15 is a figure for explaining an example of generating processingof weighted parallax distribution to which weight information isapplied;

FIG. 16 is a figure for explaining an example of generating processingof weighted parallax distribution to which weight information isapplied;

FIG. 17 is a figure for explaining an example of generating processingof weighted parallax distribution to which weight information isapplied;

FIGS. 18A and 18B are figures for explaining relationship data of aparallax d and a weighted parallax distribution H(d) and relationshipdata of parallax d and parallax accumulative distribution O(d);

FIG. 19 is a figure illustrating a flowchart for explaining an exampleof processing of virtual view point position determining processingsequence executed by an image processing apparatus;

FIG. 20 is a figure illustrating a flowchart for explaining an exampleof processing of virtual view point position determining processingsequence executed by an image processing apparatus;

FIGS. 21A and 21B are figures for explaining an example of determiningprocessing virtual view point positions having virtual view pointintervals of non-regular intervals;

FIG. 22 is a figure illustrating a flowchart explaining a processingsequence of an example of processing for generating virtual view pointimages using shift processing;

FIG. 23 is a figure for explaining relationship data of a parallax d anda weighted parallax distribution H(d) and relationship data of parallaxd and parallax accumulative distribution O(d);

FIG. 24 is a figure illustrating a flowchart for explaining an exampleof processing of virtual view point position determining processingsequence executed by an image processing apparatus;

FIG. 25 is a figure explaining an example of a weighted parallaxdistribution and a corrected weighted parallax distribution in whichcorrection of parallel movement has been done;

FIG. 26 is a figure for explaining an example of processing of parallelmovement of a parallax distribution;

FIG. 27 is a figure for explaining parallel movement processing of animage with regard to an example of virtual view point image generatingprocessing; and

FIG. 28 is a figure for explaining an example of configuration of animage processing apparatus.

DETAILED DESCRIPTION OF THE EMBODIMENTS

Embodiments of an image processing apparatus, an image processingmethod, and a program according to an embodiment of the presentdisclosure will be explained in detail with reference to drawings.Explanation will be made according to the following items.

1. (First Embodiment) Embodiment in which determining processing ofvirtual view point positions is executed in view of image quality

1-1. Overall processing sequence of processing executed by imageprocessing apparatus

1-2. Determining processing of virtual view point positions according tofirst embodiment

2. (Second Embodiment) Embodiment in which determining processing ofvirtual view point positions is executed on the basis of appropriateamount of parallax and image weight

3. (Third Embodiment) Embodiment in which determining processing ofvirtual view point positions is executed on the basis of image quality,appropriate amount of parallax, and image weight

4. (Fourth Embodiment) Example of processing for determining virtualview point positions of non-regular intervals

5. (Fifth Embodiment) Example of processing for generating virtual viewpoint images using shift processing

6. Example of Configuration of Image Processing Apparatus

7. Summary of Configuration of the Present Disclosure

1. First Embodiment Embodiment in which Determining Processing ofVirtual View Point Positions is Executed in View of Image Quality

First, an embodiment in which determining processing of virtual viewpoint positions is executed in view of image quality will be explainedas the first embodiment of an image processing apparatus of the presentdisclosure.

[1-1. Overall Processing Sequence of Processing Executed by ImageProcessing Apparatus]

FIG. 1 is a flowchart explaining an overall processing sequence ofprocessing executed by an image processing apparatus according to thepresent embodiment.

First, overall processing sequence of processing executed by the imageprocessing apparatus according to the present embodiment will beexplained with reference to the flow of FIG. 1, and thereafter, detailsof processing of each step will be explained in order.

In step S101, the image processing apparatus receives an original lefteye image (L image) and an original right eye image (R image) forthree-dimensional image display, and obtains parallax information usingthese received LR images.

In other words, parallax information is obtained by using standard LRimages with which an optimum three-dimensional image is observed whenobserved from a standard visual position with respect to a displaydisplaying a three-dimensional (3D) image.

The parallax information corresponds to a displacement between images ofthe same subject included in standard LR images (pixel displacement in ahorizontal direction), and is information corresponding to a distance ofa subject. More specifically, for example, data having parallaxinformation (subject distance information) in units of pixels aregenerated.

In step S102, virtual view point positions of multi-view point images tobe generated are determined on the basis of the parallax informationobtained in step S101.

The standard LR images have a certain range of parallax from a largeparallax to a small parallax. In step S102, the virtual view pointpositions of the multi-view point images to be generated are determinedon the basis of this parallax distribution information, and the like.

In step S103, the multi-view point images including the virtual viewpoint images corresponding to the virtual view point positionsdetermined in step S102 are generated.

The virtual view point images are generated using the received standardLR images.

Finally, in step S104, image display processing is executed using thevirtual view point images generated in step S103.

As described above, the received LR images can be observed as an optimumthree-dimensional image when it is observed from a standard observationposition, but when the standard LR images are observed when theobservation position is displaced from the standard position, reversedvision or crosstalk occurs.

However, the reversed vision and the crosstalk can be prevented byallowing the observer's left eye and right eye to observe two LR imagesselected from the multi-view point images generated according to thisprocessing, in accordance with the observation position.

In step S104, this kind of image display is executed.

Subsequently, the details of processing of each step in the flowchart asillustrated in FIG. 1 will be explained.

(Step S101: Acquisition of Parallax Information)

First, acquisition processing of the parallax information in step S101will be explained.

In step S101, an original left eye image (L image) and an original righteye image (R image) for three-dimensional image display are received,and parallax information is obtained using these received LR images. Asdescribed above, the parallax information corresponds to a displacementbetween images of the same subject included in standard LR images (pixeldisplacement in a horizontal direction), and is informationcorresponding to a distance of a subject. More specifically, forexample, data having parallax information (subject distance information)in units of pixels are generated.

This acquisition of the parallax information is executed according to,for example, an existing method as follows.

(a) block matching-based parallax information acquisition processing

(b) DP (dynamic programming) matching-based parallax informationacquisition processing

(c) segmentation-based parallax information acquisition processing

(d) learning-based parallax information acquisition processing

(e) Parallax information acquisition processing of a combination of theabove methods

For example, the parallax information is obtained according to any oneof the above methods (a) to (e).

The block matching-based parallax information acquisition processingwill be briefly explained.

In the received original left eye image (L image) the received originalright eye image (R image), for example, a pixel region (block) of the Limage is selected, and a block similar to the selected block is detectedfrom the R image. In other words, blocks (matching blocks) determined tobe regions obtained by shooting the same subject are selected from theLR images. Further, position displacement of the matching blocks betweenthe LR images (the number of pixels in the horizontal direction and thelike) is measured.

The position displacement of the blocks is different according to thedistance of the subject taken in that block.

In other words, the position displacement of the blocks corresponds tothe subject distance, and this position displacement information isobtained as the parallax information.

It should be noted that an example of form of expression of thisparallax information includes a depth map (distance image). The depthmap is an image which expresses, for example, parallax between the Limage and the R image in units of pixels (subject distance) asbrightness in units of pixels. In the depth map, for example, a brightregion indicates a close subject (close to a camera), and a dark regionindicates a far subject (far from the camera). In other words, the depthmap is an image in which the subject distance is represented as thebrightness.

In step S101, for example, this kind of depth map is generated as theparallax information.

(Step S102: Determination of Virtual View Point Positions)

The determining processing of the virtual view point positions in stepS102 will be explained in detail later.

(Step S103: Generation of Virtual View Point Image)

Subsequently, the generating processing of the virtual view point imagesin step S103 will be explained.

In step S103, images corresponding to images observed from the virtualview point positions determined in step S102 are generated. In otherwords, the virtual view point images are generated. In step S102, forexample, a predetermined number of (for example, 10) virtual view pointsare determined, and in step S103, virtual view point imagescorresponding to the virtual view points are generated.

The virtual view point images are generated using the received standardLR images. In other words, they are generated using the original lefteye image (L image) and the original right eye image (R image) forthree-dimensional image display.

Specific example of generating processing of virtual view point imageswill be explained with reference to FIG. 2.

FIG. 2 shows an original left eye image (L image) 101 and an originalright eye image (R image) 102 which are received by the image processingapparatus, and also shows a virtual view point image 103 generated basedon these LR images.

The left eye image (L image) 101 is an image observed from the left eyeview point position at the standard position, and the right eye image (Rimage) 102 is an image observed from the right eye view point positionat the standard position.

The view point position of the left eye image (L image) 101 is definedas 0.0, and the view point position of the right eye image (R image) 102is defined as 1.0.

FIG. 2 illustrates an example of processing where, for example, anobservation image from a view point position of 0.3, which is betweenview point positions of 0.0 to 1.0, is generated as a virtual view pointimage 103.

In the left eye image (L image) 101 and the right eye image (R image)102, the same subject (apple) is taken at respectively differentpositions. In the L image and the R image, the positions of the samesubject are at different positions because their view point positionsare different.

When the virtual view point image 103 observed from the view pointposition of 0.3, which is between the view point positions of 0.0 to1.0, is generated, the position of this subject (apple) is set by linearinterpolation. By changing the subject position along line L1 asillustrated in FIG. 2, the virtual view point images can be generated bydetermining the subject positions of the virtual view point images atthe respective virtual view points.

As described above, the virtual view point images at the virtual viewpoint positions are generated by linear interpolation processing basedon the received LR images.

When a virtual view point image is generated, the virtual view pointimage can be generated by processing of blending the two images usingboth of the received LR images.

Alternatively, a virtual view point image can be generated using onlyone image by means of processing of displacing the subject positionaccording to the virtual view point position using only the L image oronly the R image.

Alternatively, processing may be performed as follows. At a virtual viewpoint position close to the L image side, a virtual view point image maybe generated using only the L image. At a virtual view point positionclose to the R image side, a virtual view point image may be generatedusing only the R image.

(S104: Image Display Processing)

Subsequently, the processing of step S104, i.e., the details of theimage display processing using the virtual view point images generatedin step S103, will be explained with reference to FIG. 3.

A display image generated by an image processing apparatus according toan embodiment of the present disclosure is a display image of a nakedeye 3D display apparatus, with which a user can view a stereoscopicimage without wearing glasses.

The naked eye 3D display apparatus includes a lenticular sheet or aparallax barrier (parallax barrier) on a display surface, which controlsimages entering into the left eye and the right eye in accordance with aviewing/listening position. In other words, this has such configurationthat a left eye image and a right eye image are generated, and the lefteye image is allowed to be observed with only the left eye, and theright eye image is allowed to be observed with only the right eye.

Using this kind of technique, crosstalk in which images entering intothe left eye and the right eye are mixed is reduced, and this enablesstereoscopic vision without wearing glasses.

As a result of generation of the virtual view point images in step S103,the multi-view point images made of multiple view points (for example, Nview points) including the received LR images are generated.

These N images are displayed on the naked eye 3D display apparatus, andin accordance with the observation position of the observer, differentcombinations of view point images are respectively perceived by the lefteye and the right eye of the observer, so that the optimum 3D imagedisplay is executed according to the observation position.

FIG. 3 illustrates an example of display processing using first viewpoint to fourth view point images where the number N of multi-view pointimages including the received LR images is 4.

FIG. 3 (a) multi-view point images illustrate multi-view point imagesincluding the received LR images.

First, an image is generated by interleaving these four view pointimages.

This is FIG. 3 (b) interleaved image.

For example, a first view point image to a fourth view point image arearranged in the horizontal direction, and the interleaved image isgenerated.

In the interleaved image, barriers according to the observationdirections are set, so that when observed from a certain direction, onlya particular view point image is configured to be observed.

FIG. 3 (c) observation image example illustrates an example of imageobserved by either the left eye or the right eye of the observer from acertain observation position.

This example is a barrier setting in which the second view point imageis observed.

In order to perceive this as a three-dimensional image, it is necessaryfor the left eye and the right eye of the observer to respectivelyperceive observation images from different view point positions.

More specifically, for example, as illustrated in FIG. 4, this may be asetting as follows: the second view point image is perceived as (c1)left eye view point image, and the fourth view point image is perceivedas (c2) right eye view point image.

The naked eye 3D display apparatus using the barrier method performsdisplay processing so that the barrier and the observation image are indifferent settings in accordance with the observation position of theobserver as described above.

In the lenticular method, a pair of different view point images are alsorespectively observed with the left eye and the right eye of theobserver in accordance with the observation position.

In the image display processing in step S104, the 3D image displayprocessing is executed so that a pair of different view point images areobserved according to the observation position of the observer asdescribed above.

[1-2. Determining Processing of Virtual View Point Positions Accordingto First Embodiment]

Subsequently, the details of the determining processing of the virtualview point positions executed in step S102 in the flowchart asillustrated in FIG. 1 will be explained.

FIG. 5 is a flowchart for explaining a detailed sequence of thedetermining processing of the virtual view point positions executed instep S102 in the flowchart as illustrated in FIG. 1.

First, a series of processing of the determining processing of thevirtual view point positions will be explained with reference to thisflowchart.

First, in step S121, the quality of the virtual view point image isestimated. More specifically, an image quality evaluation value (Q) iscalculated.

Subsequently, in step S122, a virtual view point interval (G) isdetermined according to the image quality (image quality evaluationvalue (Q)) of the virtual view point images obtained in step S121.

Finally, in step S123, the virtual view point positions are determinedaccording to the virtual view point interval (G) determined in stepS122.

Hereinafter, the details of each of the above processing will beexplained with reference to the drawings.

(S121: Estimating Processing of Image Quality)

First, calculation processing of the qualities of the virtual view pointimages (image quality evaluation value (Q)) executed in step S121 willbe explained with reference to FIG. 6 and the like.

The virtual view point images are obtained by executing the linearinterpolation processing and the like using the received LR images asexplained with reference to FIG. 2 above.

In other words, usable images are only two images of the received lefteye image (L image) and the received right eye image (R image).

When the virtual view point images are generated with such settings, theimage quality of the generated virtual view point images has thefollowing tendency

(a) “A case where parallax detection fails” or “a case where theprecision of the parallax detection is low”, the quality of thegenerated virtual view point videos is reduced.

(b) In a case of “a complicated image” or “an image in which occlusionoccurs much”, the quality of the generated virtual view point videos isreduced.

Like FIG. 2 explained above, FIG. 6 shows an original left eye image (Limage) 121 and an original right eye image (R image) 122 which arereceived by the image processing apparatus, and also shows a virtualview point image 123 generated based on these LR images.

The view point position of the left eye image (L image) 121 is definedas 0.0, and the view point position of the right eye image (R image) 122is defined as 1.0.

FIG. 6 illustrates an example of processing where, for example, anobservation image from a view point position of 0.3, which is betweenview point positions of 0.0 to 1.0, is generated as a virtual view pointimage 123.

FIG. 6 (a) illustrates an example of processing performed on a notcomplicated image. FIG. 6 (b) illustrates an example of processingperformed on a complicated image.

In general, when the complexity of an image increases, the qualities ofthe virtual view point images is reduced.

For example, the following method is applied as an example of estimatingmethod of the qualities of the virtual view point images.

(A) Estimating processing of the virtual view point image quality inaccordance with the degree of reliability of parallax information

(B) Estimating processing of the virtual view point image quality basedon comparison result between the virtual view point images generatedbased on the received images and the received images

(C) Generating actually used virtual view point images, and qualityestimating processing based on the generated images

For example, the quality evaluation value Q of the virtual view pointimages is calculated with any one of the above processing (A) to (C) ora combination of the above processing (A) to (C).

Specific example of the processing will be explained.

(A) Estimating processing of the virtual view point image quality inaccordance with the degree of reliability of parallax information

First, the estimating processing of the virtual view point image qualityin accordance with the degree of reliability of parallax informationwill be explained.

This processing is processing in which the degree of reliability of theparallax information obtained in step S101 as illustrated in the flow ofFIG. 1 (for example, depth map) is calculated, and this parallaxinformation reliability degree is defined as the image quality.

As described above, the parallax information is generated based on thereceived LR images, and for example, the parallax information is datagenerated by processing such as block matching and the like. In somecases, error may occur in this processing, and when this parallaxinformation is incorrect, the quality of the generated virtual viewpoint image is reduced.

As described above, the image quality of the virtual view point image isclosely related to the precision of the parallax information. Therefore,the precision of the parallax information is calculated, and this can bedefined as the image qualities of the virtual view point images.

A specific example of calculation method will be explained withreference to FIGS. 7A and 7B.

FIGS. 7A and 7B illustrate an example of processing in which quadraticdifferentials of the parallax values obtained from the received LRimages are integrated, and this integration value is defined as thedegree of reliability of the parallax information, and determination ismade as follows: the smaller the integration value is, the higher thedegree of reliability of the parallax information is determined to be.

The parallax obtained from the LR images represents a distributionaccording to the subject distance, but for example, when a subject of acertain distance exists, the value of the parallax corresponding to thissingle subject is constant where the parallax information is calculatedcorrectly. However, when it is not correctly calculated, various valuesoccur. In other words, the parallax value varies.

More specifically, when the parallax is correctly obtained, theinfluence caused by noises and the like is low, and the parallax valuetends to become smooth as illustrated in FIG. 7A, and the integrationvalue of the quadratic differentials of the parallax values decreases.In such case, the parallax information reliability degree is determinedto be high, and as a result, the image qualities of the virtual viewpoint images is also determined to be high (i.e., the quality evaluationvalue Q of the virtual view point images is high).

On the other hand, when the parallax is not correctly obtained, theparallax values are dispersed to various values and tend to vary due tothe influence of the noise and the like as illustrated in FIG. 7B, andaccordingly, the integration value of the quadratic differentials of theparallax values increases. In such case, the parallax informationreliability degree is determined to be low, and as a result, the imagequalities of the virtual view point images is also determined to be low(i.e., the quality evaluation value Q of the virtual view point imagesis low).

It should be noted that the reliability degree determining processing ofthe parallax information is not limited to such processing. Anotherexample of method includes calculating correlation between a distanceimage including parallax information (depth map) and original images andcalculating the correlation value as the degree of reliability. Suchmethod may also be applied.

(B) Estimating processing of the virtual view point image quality basedon comparison result between the virtual view point images generatedbased on the received images and the received images

Subsequently, a specific example of estimating processing of the virtualview point image quality based on comparison result between the virtualview point images generated based on the received images and thereceived images will be explained with reference to FIG. 8.

The processing of (B) is processing in which, for example, a virtualview point image R′ corresponding to the R image from a view pointposition of 1.0 is generated using the received left eye image (L image)from a view point position of 0.0, and the generated virtual view pointimage R′ and the received right eye image (R image), i.e., receivedimage, are compared, and according to the difference, the quality of thevirtual view point image is determined.

In other words, when the difference is large, the quality of the virtualview point image is determined to be low, and when the difference issmall, the quality of the virtual view point image is determined to behigh.

Likewise, a virtual view point image L′ corresponding to the L imagefrom a view point position of 0.0 is generated using the received righteye image (R image) from a view point position of 1.0, and the generatedvirtual view point image L′ and the received left eye image (L image),i.e., received image, are compared, and according to the difference, thequality of the virtual view point image is determined.

FIG. 8 shows an example of processing as follows.

Step 1: processing of generating the virtual view point image R′corresponding to the R image of a view point position of 1.0 using theparallax information (depth map) and the received left eye image (Limage) of a view point position of 0.0

Step 2: processing of comparing the generated virtual view point imageR′ and the received right eye image (R image), i.e., received image

In the comparing processing of step 2, for example, a summation ofdifferences between the two images is calculated, and when thedifference is smaller, the virtual view point image R′ is determined tobe a more correct image, and the degree of reliability is determined tobe high (=the quality evaluation value Q of the virtual view point imageis determined to be high), and when the difference is larger, thevirtual view point image R′ is determined to be a more incorrect image,and the degree of reliability is determined to be low (=the qualityevaluation value Q of the virtual view point image is determined to below).

The quality evaluation value Q of the virtual view point image can becalculated according to this kind of image comparison.

(Step S122: Determining Processing of Virtual View Point Interval)

Subsequently, the processing of step S122, i.e., a specific example ofprocessing for determining the virtual view point interval (G) inaccordance with the image qualities of the virtual view point images(image quality evaluation value (Q)) obtained in step S121, will beexplained.

In this processing, the virtual view point interval (G) is determined byapplying the following parameters.

Q: virtual view point image quality evaluation value (value calculatedin step S121)

Q′: virtual view point interval calculation parameter (user input)

By applying these parameters, the virtual view point interval (G) isdetermined.

On the basis of these parameters Q, Q′, the virtual view point interval(G) is calculated by the following expression.G=Q/Q′

Where N is the total number of virtual view points, Qmax is the maximumvalue of the quality evaluation value of the image, Q′=Qmax*N holds,then the following expression holds. When the virtual view point imagequality evaluation value Q is at the maximum value Qmax, G=Q/Q′=1/Nholds. For example, where the view point positions of the received LRimages are as follows: L image view point position=0.0, R image viewpoint position=1.0, then, the virtual view point positions are positionsobtained by dividing the length between the view point position 0.0 andthe view point position 1.0 by N.

A value set in advance or a user input value is applied as the totalnumber (N) of virtual view points, in accordance with, for example, thedisplay apparatus (display).

(Step S123: The Determining Processing of the Virtual View PointPositions)

Subsequently, the processing of step S123, i.e., a specific example ofprocessing for determining the virtual view point positions inaccordance with the virtual view point interval (G) obtained in stepS122, will be explained.

First, the parameters applied in determining the virtual view pointpositions will be explained. The parameters applied in determining thevirtual view point positions include the following parameters.

N: the total number of virtual view points

P(i): the i-th virtual view point positions [i=0, 1, 2, . . . N−2, N−1]

G: virtual view point interval

Nfix: the number of virtual view point position set as the referenceposition (=reference virtual view point position)

Pfix: the virtual view point position set as the reference position(=reference virtual view point position)

It should be noted that the following relational expression holds as arelational expression of the above parameters.P(i)=Pfix+(1−Nfix)×G

Among the above parameters, a value set in advance or a user input valueis applied as the total number (N) of virtual view points, in accordancewith, for example, the display apparatus (display).

The virtual view point position P (i) is virtual view point positioninformation determined by the processing of this step S123.

The virtual view point interval (G) is a value determined in theprocessing of step S122 explained above.

The reference virtual view point position (Pfix) and the number thereof(Nfix) are values that can be freely set. For example, the referencevirtual view point position (Pfix) and the number thereof (Nfix) areuser input values.

It should be noted that, for example, the virtual view point positionnumber at the left end of the multiple virtual view point positions isdefined as 0, and the reference virtual view point position number(Nfix) is set as 1, 2, 3, . . . toward the right side.

Hereinafter, an example of setting of virtual view point positions wherethe reference virtual view point position (Pfix) and the number thereof(Nfix) are set as various values in the setting where the total numberof virtual view points N is 9 will be explained with reference to FIGS.9 to 13.

(Virtual View Point Position Determining Processing Example 1)

The virtual view point position determining processing example 1 will beexplained with reference to FIG. 9.

This processing example 1 is an example of determining processing ofvirtual view point positions in accordance with the following setting.

The total number of virtual view points: N=9

Reference virtual view point position number: Nfix=8

Reference virtual view point position: Pfix=1.0 (received right eyeimage (R image) view point position)

Virtual view point interval: G=4/32, 3/32, 1/32, 5/32

FIG. 9 illustrates a virtual view point position determining processingexample of the following four patterns.

(1a) N=9, Nfix=8, Pfix=1.0, G=4/32

(1b) N=9, Nfix=8, Pfix=1.0, G=3/32

(1c) N=9, Nfix=8, Pfix=1.0, G=1/32

(1d) N=9, Nfix=8, Pfix=1.0, G=5/32

This shows an example in which nine thick lines as illustrated in (1a)to (4a) are virtual view point positions, and totally nine (N=9) virtualview point positions from the left end virtual view point position P(0)to the right end virtual view point position P(8) are set with a regularinterval of interval G [(1a) G=4/32, (1b) G=3/32, (1c) G=1/32, (1d)G=5/32].

(1a) is an example where N=9, Nfix=8, Pfix=1.0, G=4/32.

First, the virtual view point P(8) of the reference virtual view pointposition number Nfix=8 is set at the reference virtual view pointposition Pfix=1.0, in accordance with the following setting condition:the reference virtual view point position Pfix=1.0 (received right eyeimage (R image) view point position) and the reference virtual viewpoint position number Nfix=8.

Subsequently, the remaining virtual view points P(0) to P(7) are setwith a virtual view point interval G=4/32 from the reference position atwhich the virtual view point P(8) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (1a) are set.

(1b) is an example where N=9, Nfix=8, Pfix=1.0, G=3/32.

First, the virtual view point P(8) of the reference virtual view pointposition number Nfix=8 is set at the reference virtual view pointposition Pfix=1.0, in accordance with the following setting condition:the reference virtual view point position Pfix=1.0 (received right eyeimage (R image) view point position) and the reference virtual viewpoint position number Nfix=8.

Subsequently, the remaining virtual view points P(0) to P(7) are setwith a virtual view point interval G=3/32 from the reference position atwhich the virtual view point P(8) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (1b) are set.

(1c) is an example where N=9, Nfix=8, Pfix=1.0, G=1/32.

First, the virtual view point P(8) of the reference virtual view pointposition number Nfix=8 is set at the reference virtual view pointposition Pfix=1.0, in accordance with the following setting condition:the reference virtual view point position Pfix=1.0 (received right eyeimage (R image) view point position) and the reference virtual viewpoint position number Mix=8.

Subsequently, the remaining virtual view points P(0) to P(7) are setwith a virtual view point interval G=1/32 from the reference position atwhich the virtual view point P(8) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (1c) are set.

(1d) is an example where N=9, Nfix=8, Pfix=1.0, G=5/32.

First, the virtual view point P(8) of the reference virtual view pointposition number Nfix=8 is set at the reference virtual view pointposition Pfix=1.0, in accordance with the following setting condition:the reference virtual view point position Pfix=1.0 (received right eyeimage (R image) view point position) and the reference virtual viewpoint position number Nfix=8.

Subsequently, the remaining virtual view points P(0) to P(7) are setwith a virtual view point interval G=5/32 from the reference position atwhich the virtual view point P(8) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (1d) are set.

(Virtual View Point Position Determining Processing Example 2)

Subsequently, the virtual view point position determining processingexample 2 will be explained with reference to FIG. 10.

This processing example 2 is an example of determining processing ofvirtual view point positions in accordance with the following setting.

The total number of virtual view points: N=9

Reference virtual view point position number: Nfix=0

Reference virtual view point position: Pfix=0.0 (received left eye image(L image) view point position)

Virtual view point interval: G=4/32, 3/32, 1/32, 5/32

FIG. 10 illustrates a virtual view point position determining processingexample of the following four patterns.

(2a) N=9, Nfix=0, Pfix=0.0, G=4/32

(2b) N=9, Nfix=0, Pfix=0.0, G=3/32

(2c) N=9, Nfix=0, Pfix=0.0, G=1/32

(2d) N=9, Nfix=0, Pfix=0.0, G=5/32

This shows an example in which nine thick lines as illustrated in (2a)to (2d) are virtual view point positions, and totally nine (N=9) virtualview point positions from the left end virtual view point position P(0)to the right end virtual view point position P(8) are set with a regularinterval of interval G [(2a) G=4/32, (2b) G=3/32, (2c) G=1/32, (2d)G=5/32].

(2a) is an example where N=9, Nfix=0, Pfix=0.0, G=4/32.

First, the virtual view point P(0) of the reference virtual view pointposition number Nfix=0 is set at the reference virtual view pointposition Pfix=0.0, in accordance with the following setting condition:the reference virtual view point position Pfix=0.0 (received left eyeimage (L image) view point position) and the reference virtual viewpoint position number Nfix=0.

Subsequently, the remaining virtual view points P(1) to P(8) are setwith a virtual view point interval G=4/32 from the reference position atwhich the virtual view point P(0) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (2a) are set.

(2b) is an example where N=9, Nfix=0, Pfix=0.0, G=3/32.

First, the virtual view point P(0) of the reference virtual view pointposition number Nfix=0 is set at the reference virtual view pointposition Pfix=0.0, in accordance with the following setting condition:the reference virtual view point position Pfix=0.0 (received left eyeimage (L image) view point position) and the reference virtual viewpoint position number Nfix=0.

Subsequently, the remaining virtual view points P(1) to P(8) are setwith a virtual view point interval G=3/32 from the reference position atwhich the virtual view point P(0) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (2b) are set.

(2c) is an example where N=9, Nfix=0, Pfix=0.0, G=1/32.

First, the virtual view point P(0) of the reference virtual view pointposition number Nfix=0 is set at the reference virtual view pointposition Pfix=0.0, in accordance with the following setting condition:the reference virtual view point position Pfix=0.0 (received left eyeimage (L image) view point position) and the reference virtual viewpoint position number Nfix=0.

Subsequently, the remaining virtual view points P(1) to P(8) are setwith a virtual view point interval G=1/32 from the reference position atwhich the virtual view point P(0) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (2c) are set.

(2d) is an example where N=9, Nfix=0, Pfix=0.0, G=5/32.

First, the virtual view point P(0) of the reference virtual view pointposition number Nfix=0 is set at the reference virtual view pointposition Pfix=0.0, in accordance with the following setting condition:the reference virtual view point position Pfix=0.0 (received left eyeimage (L image) view point position) and the reference virtual viewpoint position number Nfix=0.

Subsequently, the remaining virtual view points P(1) to P(8) are setwith a virtual view point interval G=5/32 from the reference position atwhich the virtual view point P(0) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (2d) are set.

(Virtual View Point Position Determining Processing Example 3)

Subsequently, the virtual view point position determining processingexample 3 will be explained with reference to FIG. 11.

This processing example 3 is an example of determining processing ofvirtual view point positions in accordance with the following setting.

The total number of virtual view points: N=9

Reference virtual view point position number: Nfix=4

Reference virtual view point position: Pfix=0.0 (received left eye image(L image) view point position)

Virtual view point interval: G=4/32, 3/32, 1/32, 5/32

FIG. 11 illustrates a virtual view point position determining processingexample of the following four patterns.

(3a) N=9, Nfix=4, Pfix=0.0, G=4/32

(3b) N=9, Nfix=4, Pfix=0.0, G=3/32

(3c) N=9, Nfix=4, Pfix=0.0, G=1/32

(3d) N=9, Nfix=4, Pfix=0.0, G=5/32

This shows an example in which nine thick lines as illustrated in (3a)to (3a) are virtual view point positions, and totally nine (N=9) virtualview point positions from the left end virtual view point position P(0)to the right end virtual view point position P(8) are set with a regularinterval of interval G [(3a) G=4/32, (3b) G=3/32, (3c) G=1/32, (3d)G=5/32].

(3a) is an example where N=9, Nfix=4, Pfix=0.0, G=4/32.

First, the virtual view point P(4) of the reference virtual view pointposition number Nfix=4 is set at the reference virtual view pointposition Pfix=0.0, in accordance with the following setting condition:the reference virtual view point position Pfix=0.0 (received left eyeimage (L image) view point position) and the reference virtual viewpoint position number Nfix=4.

Subsequently, the remaining virtual view points P(0) to P(3) and P(5) toP(8) are set with a virtual view point interval G=4/32 from thereference position at which the virtual view point P(4) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (3a) are set.

(3b) is an example where N=9, Nfix=4, Pfix=0.0, G=3/32.

First, the virtual view point P(4) of the reference virtual view pointposition number Nfix=4 is set at the reference virtual view pointposition Pfix=0.0, in accordance with the following setting condition:the reference virtual view point position Pfix=0.0 (received left eyeimage (L image) view point position) and the reference virtual viewpoint position number Nfix=4.

Subsequently, the remaining virtual view points P(0) to P(3) and P(5) toP(8) are set with a virtual view point interval G=3/32 from thereference position at which the virtual view point P(4) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (3b) are set.

(3c) is an example where N=9, Nfix=4, Pfix=0.0, G=1/32. First, thevirtual view point P(4) of the reference virtual view point positionnumber Nfix=4 is set at the reference virtual view point positionPfix=0.0, in accordance with the following setting condition: thereference virtual view point position Pfix=0.0 (received left eye image(L image) view point position) and the reference virtual view pointposition number Nfix=4.

Subsequently, the remaining virtual view points P(0) to P(3) and P(5) toP(8) are set with a virtual view point interval G=1/32 from thereference position at which the virtual view point P(4) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (3c) are set.

(3d) is an example where N=9, Nfix=4, Pfix=0.0, G=5/32.

First, the virtual view point P(4) of the reference virtual view pointposition number Nfix=4 is set at the reference virtual view pointposition Pfix=0.0, in accordance with the following setting condition:the reference virtual view point position Pfix=0.0 (received left eyeimage (L image) view point position) and the reference virtual viewpoint position number Nfix=4.

Subsequently, the remaining virtual view points P(0) to P(3) and P(5) toP(8) are set with a virtual view point interval G=5/32 from thereference position at which the virtual view point P(4) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (3d) are set.

(Virtual View Point Position Determining Processing Example 4)

Subsequently, the virtual view point position determining processingexample 4 will be explained with reference to FIG. 12.

This processing example 4 is an example of determining processing ofvirtual view point positions in accordance with the following setting.

The total number of virtual view points: N=9

Reference virtual view point position number: Nfix=2

Reference virtual view point position: Pfix=0.0 (received left eye image(L image) view point position)

Virtual view point interval: G=4/32, 3/32, 1/32, 5/32

FIG. 12 illustrates a virtual view point position determining processingexample of the following four patterns.

(4a) N=9, Nfix=2, Pfix=0.0, G=4/32

(4b) N=9, Nfix=2, Pfix=0.0, G=3/32

(4c) N=9, Nfix=2, Pfix=0.0, G=1/32

(4d) N=9, Nfix=2, Pfix=0.0, G=5/32

This shows an example in which nine thick lines as illustrated in (4a)to (4d) are virtual view point positions, and totally nine (N=9) virtualview point positions from the left end virtual view point position P(0)to the right end virtual view point position P(8) are set with a regularinterval of interval G [(4a) G=4/32, (4b) G=3/32, (4c) G=1/32, (4d)G=5/32].

(4a) is an example where N=9, Nfix=2, Pfix=0.0, G=4/32.

First, the virtual view point P(2) of the reference virtual view pointposition number Nfix=2 is set at the reference virtual view pointposition Pfix=0.0, in accordance with the following setting condition:the reference virtual view point position Pfix=0.0 (received left eyeimage (L image) view point position) and the reference virtual viewpoint position number Nfix=2.

Subsequently, the remaining virtual view points P(0) to P(1) and P(3) toP(8) are set with a virtual view point interval G=4/32 from thereference position at which the virtual view point P(2) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (4a) are set.

(4b) is an example where N=9, Nfix=2, Pfix=0.0, G=3/32.

First, the virtual view point P(2) of the reference virtual view pointposition number Nfix=2 is set at the reference virtual view pointposition Pfix=0.0, in accordance with the following setting condition:the reference virtual view point position Pfix=0.0 (received left eyeimage (L image) view point position) and the reference virtual viewpoint position number Nfix=2.

Subsequently, the remaining virtual view points P(0) to P(1) and P(3) toP(8) are set with a virtual view point interval G=3/32 from thereference position at which the virtual view point P(2) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (4b) are set.

(4c) is an example where N=9, Nfix=2, Pfix=0.0, G=1/32.

First, the virtual view point P(2) of the reference virtual view pointposition number Nfix=2 is set at the reference virtual view pointposition Pfix=0.0, in accordance with the following setting condition:the reference virtual view point position Pfix=0.0 (received left eyeimage (L image) view point position) and the reference virtual viewpoint position number Nfix=2.

Subsequently, the remaining virtual view points P(0) to P(1) and P(3) toP(8) are set with a virtual view point interval G=1/32 from thereference position at which the virtual view point P(2) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (4c) are set.

(4d) is an example where N=9, Nfix=2, Pfix=0.0, G=5/32.

First, the virtual view point P(2) of the reference virtual view pointposition number Nfix=2 is set at the reference virtual view pointposition Pfix=0.0, in accordance with the following setting condition:the reference virtual view point position Pfix=0.0 (received left eyeimage (L image) view point position) and the reference virtual viewpoint position number Nfix=2.

Subsequently, the remaining virtual view points P(0) to P(1) and P(3) toP(8) are set with a virtual view point interval G=5/32 from thereference position at which the virtual view point P(2) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (4d) are set.

(Virtual View Point Position Determining Processing Example 5)

Subsequently, the virtual view point position determining processingexample 5 will be explained with reference to FIG. 13.

This processing example 5 is an example of determining processing ofvirtual view point positions in accordance with the following setting.

The total number of virtual view points: N=9

Reference virtual view point position number: Nfix=4

Reference virtual view point position: Pfix=0.5 (view point position atthe middle of the received left eye image (L image) and the receivedright eye image (R image))

Virtual view point interval: G=4/32, 3/32, 1/32, 5/32

FIG. 13 illustrates a virtual view point position determining processingexample of the following four patterns.

(5a) N=9, Nfix=4, Pfix=0.5, G=4/32

(5b) N=9, Nfix=4, Pfix=0.5, G=3/32

(5c) N=9, Nfix=4, Pfix=0.5, G=1/32

(5d) N=9, Nfix=4, Pfix=0.5, G=5/32

This shows an example in which nine thick lines as illustrated in (5a)to (5d) are virtual view point positions, and totally nine (N=9) virtualview point positions from the left end virtual view point position P(0)to the right end virtual view point position P(8) are set with a regularinterval of interval G [(5a) G=4/32, (5b) G=3/32, (5c) G=1/32, (5d)G=5/32].

(5a) is an example where N=9, Nfix=4, Pfix=0.5, G=4/32.

First, the virtual view point P(4) of the reference virtual view pointposition number Nfix=4 is set at the reference virtual view pointposition Pfix=0.5, in accordance with the following setting condition:the reference virtual view point position Pfix=0.5 (view point positionat the middle of the received left eye image (L image) and the receivedright eye image (R image)) and the reference virtual view point positionnumber Nfix=4.

Subsequently, the remaining virtual view points P(0) to P(3) and P(5) toP(8) are set with a virtual view point interval G=4/32 from thereference position at which the virtual view point P(4) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (5a) are set.

(5b) is an example where N=9, Nfix=4, Pfix=0.5, G=3/32.

First, the virtual view point P(4) of the reference virtual view pointposition number Nfix=4 is set at the reference virtual view pointposition Pfix=0.5, in accordance with the following setting condition:the reference virtual view point position Pfix=0.5 (view point positionat the middle of the received left eye image (L image) and the receivedright eye image (R image)) and the reference virtual view point positionnumber Nfix=4.

Subsequently, the remaining virtual view points P(0) to P(3) and P(5) toP(8) are set with a virtual view point interval G=3/32 from thereference position at which the virtual view point P(4) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (5b) are set.

(5c) is an example where N=9, Nfix=4, Pfix=0.5, G=1/32.

First, the virtual view point P(4) of the reference virtual view pointposition number Nfix=4 is set at the reference virtual view pointposition Pfix=0.5, in accordance with the following setting condition:the reference virtual view point position Pfix=0.5 (view point positionat the middle of the received left eye image (L image) and the receivedright eye image (R image)) and the reference virtual view point positionnumber Nfix=4.

Subsequently, the remaining virtual view points P(0) to P(3) and P(5) toP(8) are set with a virtual view point interval G=1/32 from thereference position at which the virtual view point P(4) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (5c) are set.

(5d) is an example where N=9, Nfix=4, Pfix=0.5, G=5/32.

First, the virtual view point P(4) of the reference virtual view pointposition number Nfix=4 is set at the reference virtual view pointposition Pfix=0.5, in accordance with the following setting condition:the reference virtual view point position Pfix=0.5 (view point positionat the middle of the received left eye image (L image) and the receivedright eye image (R image)) and the reference virtual view point positionnumber Nfix=4.

Subsequently, the remaining virtual view points P(0) to P(3) and P(5) toP(8) are set with a virtual view point interval G=5/32 from thereference position at which the virtual view point P(4) is set.

With this processing, nine virtual view point positions P(0) to P(8) asillustrated in (5d) are set.

2. Second Embodiment Embodiment in which Determining Processing ofVirtual View Point Positions is Executed on the Basis of AppropriateAmount of Parallax and Image Weight

Subsequently, an embodiment in which determining processing of virtualview point positions is executed on the basis of appropriate amount ofparallax and an image weight will be explained as the second embodimentof an image processing apparatus of the present disclosure.

In this second embodiment, overall processing sequence is executedaccording to the flow as illustrated in FIG. 1 explained in the firstembodiment above.

The determining processing of the virtual view point positions executedin step S102 in the flowchart as illustrated in FIG. 1 is processingdifferent from the first embodiment explained above.

FIG. 14 is a flowchart for explaining an example of detailed sequence ofvirtual view point position determining processing according to thissecond embodiment.

First, the virtual view point position determining processing sequenceaccording to the present embodiment will be explained with reference tothe flow of FIG. 14, and thereafter, details of processing of each stepwill be explained in order.

First, an image processing apparatus determines an appropriate amount ofparallax in step 141. Although specific processing will be explainedlater, the appropriate amount of parallax is determined in view of, forexample, fusional parallax amount or crosstalk allowable amount.

Subsequently, in step S142, weighted parallax distribution is calculatedon the basis of weight information and parallax information calculatedfrom a received left eye image (L image) and a received right eye image(R image).

The weight information is information indicating image weight in unitsof image regions included in an image, and, for example, the weightinformation is information in which larger weights are set in imageregions that are likely to attract attention of an observer, which are,more specifically, a central portion of an image and a facial imageregion of a person, the details of which will be explained later.

On the basis of the above information, the weighted parallaxdistribution information is calculated.

Subsequently, in step S143, the maximum value of parallax is calculatedby applying the weighted parallax distribution information calculated instep S142. This is a value corresponding to the maximum value ofparallax set in the virtual view point image. The details of thecalculation processing will be explained later.

Subsequently, in step S144, a virtual view point interval (G) iscalculated using the maximum value of parallax calculated in step S143and the appropriate amount of parallax determined in step S141.

Finally, in step S145, virtual view point positions are determined byapplying the virtual view point interval (G) determined in step S144.

Hereinafter, the details of each of the above processing will beexplained with reference to the drawings.

(S141: Determining Processing of the Appropriate Amount of Parallax)

First, an example of determining processing of the appropriate amount ofparallax executed in step S141 will be explained.

The appropriate amount of parallax is determined in view of, forexample, the fusional parallax amount and the crosstalk allowableamount.

The fusional parallax amount is corresponds to the limitation of theamount of parallax which allows observation of a stereoscopic imagewhere the LR images set with the parallax are observed, and theappropriate amount of parallax can be set as an amount of parallax halfthe fusional parallax amount, for example.

The fusional parallax amount is a value calculated from the size of adisplay, a distance between an observer and the display, a distancebetween both eyes, and the like. For example, this is set as a uniquevalue for the display apparatus, and a value half this value isdetermined as the appropriate amount of parallax using this fusionalparallax amount thus set.

For example, it is assumed as follows. The visual distance between theobserver and the display apparatus (display) is a standard visualdistance (=third times the height of the display). The screen size ofthe display in the horizontal direction is 1920 pixels (pixels). Thedistance between both eyes of the observer is 65 mm (average distancebetween both eyes of adults. In this case, the fusional parallax amountcan be calculated as 114 pixels (pixels), and the appropriate amount ofparallax can be calculated as 57 pixels (pixels).

The appropriate amount of parallax can also be determined in view of,for example, the crosstalk allowable amount.

The crosstalk is an error in which the left eye image is observed by theright eye, and the right eye image is observed by the left eye.

A certain crosstalk allowable amount is set, and the amount of parallaxwith which the crosstalk is equal to or less than the crosstalkallowable amount is set as the appropriate amount of parallax.

As described above, the appropriate amount of parallax is determined inview of any one of the fusional parallax amount and the crosstalkallowable amount or both of them.

(S142: Calculation of Weighted Parallax Distribution)

Subsequently, the details of the calculation processing of the weightedparallax distribution executed in step S142 will be explained.

In other words, the weighted parallax distribution is calculated on thebasis of weight information and parallax information calculated from areceived left eye image (L image) and a received right eye image (Rimage).

The weight information is information in which larger weights are set inimage regions that are likely to attract attention of an observer, whichare, more specifically, a central portion of an image and a facial imageregion of a person, the details of which will be explained later.

Multiple examples of processing (1) to (3) below for calculating theweighted parallax distribution using the weight information will beexplained with reference to FIGS. 15 to 17.

(1) Example of processing where weights are set according to imageregions (a larger weight is set in a central region)

(2) Example of processing where weights are set according to subjects (alarger weight is set in a person region)

(3) Example of processing in which both of processing of (1) and (2)explained above are combined

First, with reference to FIG. 15, in (1) example of processing whereweights are set according to image regions (a larger weight is set in acentral region), the example of processing for calculating this weightedparallax distribution will be explained.

A received image 151 as illustrated in FIG. 15 is an original left eyeimage (L image) or an original right eye image (R image). The parallaxinformation 152 is parallax information obtained using the original lefteye image (L image) and the original right eye image (R image).

A parallax histogram as illustrated in FIG. 15 (A) is a histogram of theparallax information of the parallax information 152. In the histogram,the horizontal axis denotes the value of parallax, and the vertical axisdenotes the frequency.

The histogram of FIG. 15(A) is simplified, but subjects in the imageinclude a tree, a person, and a background. The histogram of FIG. 15(A)is a histogram reflecting data corresponding to the parallaxes(corresponding to subject distances).

It is understood that the parallaxes (subject distances) are set suchthat a subject closest to a camera is the “tree”, a subject subsequentlyclose thereto is the “person”, and a subject subsequently close theretois the “background”.

In the present embodiment, the parallax information is corrected usingpredetermined weight information.

In the example as illustrated in FIG. 15, correction is executed on thebasis of weights according to image regions, i.e., weight informationset such that a higher weight is set in a central region of an image isset, and a weighted parallax distribution (parallax histogram) having adistribution as illustrated in FIG. 15 (B) is generated.

For example, weight information A 153 as illustrated in FIG. 15 is usedas the weight information in which weights are set according to imageregions.

As illustrated in the figure, the weight information A 153 is an imagein which weights according to image regions are indicated usingbrightness.

This indicates that a higher weight is set in a central region havinghigh brightness, and a lower weight is set in a peripheral region havinglow brightness.

By multiplying the parallax information 152 by this weight information A153 in units of corresponding pixels, the parallax histogram iscalculated as the weighted parallax distribution illustrated in FIG. 15(B).

As illustrated in FIG. 15 (B), the “person” and the “tree” exists in acentral region of the image, and therefore, the value of frequencyincreases, whereas most of the “background” exist in a peripheralregion, the value of frequency decreases. As described above, the imageprocessing apparatus generates, for example, weighted parallaxinformation in which weights are set according to image regions.

Subsequently, with reference to FIG. 16, in (2) example of processingwhere weights are set according to subjects (a larger weight is set in aperson region), the example of processing for calculating this weightedparallax distribution will be explained.

A received image 151 as illustrated in FIG. 16 is an original left eyeimage (L image) or an original right eye image (R image). The parallaxinformation 152 is parallax information obtained using the original lefteye image (L image) and the original right eye image (R image).

A parallax histogram as illustrated in FIG. 16 (A) is a histogram of theparallax information of the parallax information 152 like that in FIG.15. In the histogram, the horizontal axis denotes the value of parallax,and the vertical axis denotes the frequency

In the present embodiment, correction is executed this parallaxinformation on the basis of weights according to subjects, i.e., forexample, weight information set such that a higher weight is set in aperson region of an image is set, and a weighted parallax distribution(parallax histogram) having a distribution as illustrated in FIG. 16 (B)is generated.

For example, weight information B 154 as illustrated in FIG. 16 is usedas the weight information in which weights are set according to subjectregions.

As illustrated in the figure, the weight information B 154 is an imagein which weights according to subject regions are indicated usingbrightness.

In this example, a weight of a region where subject=person is high, andthis indicates that a weight of a region where subject≠person is low.

By multiplying the parallax information 152 by this weight information B154 in units of corresponding pixels, the parallax histogram iscalculated as the weighted parallax distribution illustrated in FIG. 16(B).

As illustrated in FIG. 16 (B), the value of frequency of the “person”increases, whereas the values of frequencies of the “tree” and the“background” the decrease. The image processing apparatus generateweighted parallax information in which weights are set according tosubjects as described above.

FIG. 17 is an example of processing using both of the weight informationcorresponding to image regions and weight information corresponding tosubjects explained with reference to FIG. 15.

First, the weight information A 153 including the weight informationcorresponding to the image regions is multiplied by the weightinformation B 154 corresponding to the subjects, whereby a weight imageis generated by combining two pieces of weight information, andthereafter, multiplication by the parallax information 152(multiplication of corresponding pixels) is executed.

With this processing, the parallax histogram is calculated as theweighted parallax distribution illustrated in FIG. 17 (B).

As illustrated in FIG. 17 (B), the value of frequency of the “person”increases, whereas the values of frequencies of the “tree” and the“background” decrease. The image processing apparatus generate weightedparallax information in which weights are set according to subjects asdescribed above.

Multiple examples of calculation processing of weighted parallaxdistributions have been explained with reference to FIGS. 15 to 17.

This calculation processing of the weighted parallax distributions willbe explained.

Parallax information, weight information, a weighted parallaxdistribution are expressed as follows.

D (x, y): parallax information

W (x, y): weight information

H(d): weighted parallax distribution

However, D (x, y), W (x, y) represent the values of parallax and weightat pixel position (x, y) in the image.

It should be noted that d in H(d) means the value of parallax, and H(d)means the number of pixels having the parallax d.

Under this conditional setting, the weighted parallax distribution H(d)can be obtained using the following calculation expression.H(d)=Σ_(x,y) {W(x,y)*δ(d−D(x,y))}

However, δ(x) is a function that returns 1 when x is zero and returns 0in the other cases.

In the present embodiment, the image processing apparatus calculates theweighted parallax distribution with the calculation processing accordingto the above expression in step S142 in the flowchart as illustrated inFIG. 14.

(Step S143: Calculation of the Maximum Value of Parallax)

Subsequently, the details of the calculation processing of the maximumvalue of parallax executed in step S143 will be explained.

This processing is processing in which the weighted parallaxdistribution information calculated in step S142 is applied and themaximum value of parallax is calculated, and is processing in which themaximum value of parallax set in the virtual view point image iscalculated.

The details of the calculation processing of the maximum value ofparallax (Dabsmax) will be explained.

The parameters are defined as follows.

H(d): weighted parallax distribution (calculation value in step S142)

O(d): parallax accumulative distribution (calculated from H(d))

S: summation of weighted parallax distributions (calculated from H(d))

th: threshold value applied to the maximum parallax calculationprocessing (user input value or preset value)

Dmax: the maximum value of parallax (calculated from th)

Dmin: the minimum value of parallax (calculated from th)

Dabsmax: the maximum value of parallax (absolute value) (output value)

It should be noted that O(d)=Σ_(i=min (D) to d){(i)} holds, and Dmin=O⁻¹(th) Dmax=O⁻¹ (S-th) holds.

FIG. 18A is a figure illustrating an example of data showingrelationship between the parallax d and the weighted parallaxdistribution H(d).

FIG. 18B is a figure illustrating examples of values of the followingdata, which are shown in the relationship data of the parallax d and theparallax accumulative distribution O(d).

Summation of weighted parallax distributions: S

Threshold value: th

The maximum value of parallax: Dmax

The minimum value of parallax: Dmin

Further, S-th,

Examples of the data are shown.

The maximum value of parallax (Dabsmax) which is output can becalculated by the following expression.Dabsmax=max(abs(Dmax),abs(Dmin))

In step S143, the maximum value of parallax (Dabsmax) is calculatedaccording to the above expression.

(Step S144: Calculate Virtual View Point Interval)

Subsequently, the calculation processing of the virtual view pointinterval (G) executed in step S144 will be explained.

In other words, this is processing of calculating the virtual view pointinterval (G) using the maximum value of parallax calculated in step S143and the appropriate amount of parallax determined in step S141.

The calculation processing of the virtual view point interval (G) willbe explained.

The parameters are defined as follows.

E: difference between a virtual view point image number which is inputto the left eye and a virtual view point image number which is input tothe right eye (value determined according to display method)

D_(F): the fusional parallax amount (value according to a displayapparatus (preset value))

D_(E): fusional parallax amount between adjacent virtual view points(D_(E)=D_(F)/E)

D_(C): crosstalk allowable amount between adjacent virtual view points(value according to a display apparatus (preset value))

D_(A): appropriate amount of parallax (calculation value of step S141)

Dabsmax: the maximum value of parallax (absolute value) (calculationvalue of step S143)

G: virtual view point interval (output value)

It should be noted that D_(E)=D_(F)/E and D_(A)=min (D_(C), D_(E)) hold.

At this occasion, the virtual view point interval (G) can be calculatedaccording to the following expression.G=Dabsmax/D _(A)

In step S144, the virtual view point interval (G) is calculatedaccording to the above expression.

(Step S145: Determination of Virtual View Point Position)

Subsequently, the determining processing of the virtual view pointpositions executed in step S145 will be explained.

This determining processing of the virtual view point positions is thesame processing as the processing of step S123 of the flow of FIG. 5 ofthe first embodiment explained above. More specifically, this is thesame processing as the processing explained with reference to FIGS. 9 to13.

Various settings are made, as explained with reference to FIGS. 9 to 13,in accordance with the following values: not only the virtual view pointinterval (G) calculated in step S144 but also the total number ofvirtual view points N determined according to the display apparatus(display), the reference virtual view point position (Pfix) determinedaccording to, e.g., user input, and the reference virtual view pointposition number (Nfix).

3. Third Embodiment Embodiment in which Determining Processing ofVirtual View Point Positions is Executed on the Basis of Image Quality,Appropriate Amount of Parallax, and Image Weight

Subsequently, the determining processing of the virtual view pointpositions in view of the image quality explained in the first embodimentexplained above, the determining processing of the virtual view pointpositions in view of the appropriate amount of parallax and the imageweight explained as the second embodiment, and an example of processingof a combination thereof will be explained as the third embodiment ofprocessing executed by an image processing apparatus of the presentdisclosure.

That is, this is processing for determining the virtual view pointpositions in view of all of the following items:

the image qualities of the virtual view point images,

the appropriate amount of parallax,

the image weight.

In this third embodiment, overall processing sequence is executedaccording to the flow as illustrated in FIG. 1 explained in the firstembodiment above.

The determining processing of the virtual view point positions executedin step S102 in the flowchart as illustrated in FIG. 1 is processingdifferent from the first embodiment explained above.

FIG. 19 is a flowchart for explaining an example of detailed sequence ofvirtual view point position determining processing according to thisthird embodiment.

The virtual view point position determining processing sequenceaccording to the present embodiment will be explained with reference tothe flow of FIG. 19.

The processing of steps S161 to S163 in the flow as illustrated in FIG.19 is the same processing as the processing of steps S141 to S143 in theflow as illustrated in FIG. 14 explained in the second embodiment.

In other words, in step S161, the appropriate amount of parallax isdetermined. More specifically, for example, the appropriate amount ofparallax is determined in view of, for example, fusional parallax amountor crosstalk allowable amount.

Subsequently, in step S162, weighted parallax distribution is calculatedon the basis of weight information and parallax information calculatedfrom a received left eye image (L image) and a received right eye image(R image).

This processing is the processing explained with reference to FIGS. 15to 17 above.

Subsequently, in step S163, the maximum value of parallax is calculatedby applying the weighted parallax distribution information calculated instep S162.

This processing is the same processing as the processing of steps S141to S143 in the flow as illustrated in FIG. 14 explained in the secondembodiment.

The subsequent processing in step S164 is the same processing as theprocessing of step S121 in the flow as illustrated in FIG. 5 explainedin the first embodiment.

In other words, calculation of the image qualities of the virtual viewpoint images is executed. This processing is the processing explainedwith reference to FIGS. 6 to 8 above.

In step S165 subsequent thereto, the virtual view point interval iscalculated by applying all of the following items:

the appropriate amount of parallax obtained in step S161,

the maximum value of parallax obtained in step S163, and

the image quality obtained in step S164.

This calculation processing of the virtual view point interval will beexplained.

The parameters are defined as follows.

E: difference between a virtual view point image number which is inputto the left eye and a virtual view point image number which is input tothe right eye (value determined according to display method)

D_(F): the fusional parallax amount (value according to a displayapparatus (preset value))

D_(E): fusional parallax amount between adjacent virtual view points(D_(E=D) _(F)/E)

D_(C): crosstalk allowable amount between adjacent virtual view points(value according to a display apparatus (preset value))

D_(A): appropriate amount of parallax (calculation value of step S161)

Dabsmax: the maximum value of parallax (absolute value) (calculationvalue of step S163)

G: virtual view point interval (output value)

Q: virtual view point image quality evaluation value (value calculatedin step S164)

Q′: virtual view point interval calculation parameter (user input)

It should be noted that D_(E)=D_(F)/E and D_(A)=min (D_(E), D_(E)) hold.

At this occasion, the virtual view point interval (G) can be calculatedaccording to the following expression.G={Dmax/D _(A) }*{Q/Q′}

In step S165, the virtual view point interval (G) is calculatedaccording to the above expression.

In the final step of S166, virtual view point positions are determinedby applying the virtual view point interval (G) determined in step S165.

This determining processing of the virtual view point positions is thesame processing as the processing of step S123 of the flow of FIG. 5 ofthe first embodiment explained above. More specifically, this is thesame processing as the processing explained with reference to FIGS. 9 to13.

Various settings are made, as explained with reference to FIGS. 9 to 13,in accordance with the following values: not only the virtual view pointinterval (G) calculated in step S165 but also the total number ofvirtual view points N determined according to the display apparatus(display), the reference virtual view point position (Pfix) determinedaccording to, e.g., user input, and the reference virtual view pointposition number (Nfix).

4. Fourth Embodiment Example of Processing for Determining Virtual ViewPoint Positions of Non-Regular Intervals

Subsequently, an example of processing for determining virtual viewpoint positions of a non-regular interval will be explained as thefourth embodiment executing an image processing apparatus of the presentdisclosure.

In this fourth embodiment, overall processing sequence is executedaccording to the flow as illustrated in FIG. 1 explained in the firstembodiment above.

The determining processing of the virtual view point positions executedin step S102 in the flowchart as illustrated in FIG. 1 is processingdifferent from the first embodiment explained above.

FIG. 20 is a flowchart for explaining an example of detailed sequence ofvirtual view point position determining processing according to thisfourth embodiment.

The virtual view point position determining processing sequenceaccording to the present embodiment will be explained with reference tothe flow of FIG. 20.

The processing of steps S181 to S185 is the same processing as theprocessing of steps S161 to 165 of the flow as illustrated in FIG. 19explained in the third embodiment explained above.

In step S185, the virtual view point interval is calculated by applyingall of the following items:

the appropriate amount of parallax obtained in step S181,

the maximum value of parallax obtained in step S183, and

the image quality obtained in step S184.

Subsequently, in step S186, the virtual view point interval (G) obtainedin step S185 is mapped in a non-linear manner. A specific example ofthis processing will be explained.

The parameters are defined as follows.

i: view point number

G: virtual view point interval

G(i): virtual view point interval after mapping

A function for mapping the virtual view point interval in accordancewith the view point number (i) in a non-linear manner is, for example, afunction as illustrated in FIG. 21A. The function is stored to a memoryin the image processing apparatus in advance, and this is applied.

Subsequently, in step S187, the virtual view point positions aredetermined using the virtual view point interval G(i) determined in stepS186.

This determining processing of the virtual view point positions is thesame processing as the processing of step S123 of the flow of FIG. 5 ofthe first embodiment explained above. More specifically, this is thesame processing as the processing explained with reference to FIGS. 9 to13.

Various settings are made in accordance with the following values: notonly the virtual view point interval (G(i)) calculated in step S186 butalso the total number of virtual view points N determined according tothe display apparatus (display), the reference virtual view pointposition (Pfix) determined according to, e.g., user input, and thereference virtual view point position number (Nfix).

However, in the processing explained with reference to FIGS. 9 to 13,all the virtual view point intervals are the same, but in this example,for example, as illustrated in FIG. 21B, each virtual view pointinterval is set differently. In other words, they are set as intervalsdetermined by G(i).

The example shown in FIG. 21B shows an example where N=9, Bfix=4,Pfix=0.0.

As described above, the virtual view point image generating unit of theimage processing apparatus according to the present embodimentdetermines a first virtual view point position by means of processing inview of at least one of the image qualities of the virtual view pointimages, the appropriate amount of parallax, and the image weightsaccording to image regions, determines a second virtual view pointposition of a non-regular interval by the non-linear mapping processingperformed on the determined first virtual view point position, andgenerates a virtual view point image corresponding to the determinedsecond virtual view point position of the non-regular interval.

5. Fifth Embodiment Example of Processing for Generating Virtual ViewPoint Images Using Shift Processing

Subsequently, an example of processing of generating virtual view pointimages using shift processing will be explained as the fifth embodimentof an image processing apparatus of the present disclosure.

A processing sequence of an image processing apparatus of the presentembodiment will be explained with reference to the flowchart asillustrated in FIG. 22.

In step S201, the image processing apparatus receives an original lefteye image (L image) and an original right eye image (R image) forthree-dimensional image display, and obtains parallax information usingthese received LR images.

In other words, parallax information is obtained by using standard LRimages with which an optimum three-dimensional image is observed whenobserved from a standard visual position with respect to a displaydisplaying a three-dimensional (3D) image.

Subsequently, in step S202, the appropriate amount of parallax isdetermined.

This processing is the same processing as the processing of step S141 inthe flowchart as illustrated in FIG. 14 explained as the secondembodiment above. For example, determination is made in view of thefusional parallax amount and the crosstalk allowable amount.

Subsequently, in step S203, weighted parallax distribution is calculatedon the basis of weight information and parallax information calculatedfrom a received left eye image (L image) and a received right eye image(R image).

This processing is the same processing as the processing of step S142 inthe flowchart as illustrated in FIG. 14 explained as the secondembodiment above.

The weight information is information indicating image weight in unitsof image regions included in an image, and, for example, the weightinformation is information in which larger weights are set in imageregions that are likely to attract attention of an observer, which are,more specifically, a central portion of an image and a facial imageregion of a person, the details of which will be explained later.

The weighted parallax distribution is calculated by the same processingas the processing explained with reference to FIGS. 15 to 17 above.

Subsequently, in step S204, the determining processing of the amount ofparallel movement (the amount of shift) is executed.

This amount of parallel movement is an amount of parallel movement ofparallax distribution for moving the parallax distribution in adirection in which the parallax increases or in a direction in which theparallax decreases.

The calculation processing of the amount of parallel movement of theparallax distribution will be explained.

The parameters are defined as follows.

H(d): weighted parallax distribution (obtained value in step S203)

O(d): parallax accumulative distribution (calculated from H(d))

S: summation of parallax distribution (calculated from H(d))

Davg: average value of parallax

Dcenter: median value of parallax

Dmax: the maximum value of parallax

Dmin: the minimum value of parallax

Shift

It should be noted that the following expressions hold:O(d)=Σ_(i=min (D) to d){H(i)}, Dmin=O⁻¹ (th), Dmax=O⁻¹ (S-th),Dcenter=(Dmax−Dmin)/2, Davg=average (H(d)).

FIG. 23A is a figure illustrating an example of data showingrelationship between the parallax d and the weighted parallaxdistribution H(d).

FIG. 23B is a figure illustrating examples of values of the followingdata, which are shown in the relationship data of the parallax d and theparallax accumulative distribution O(d).

Summation of weighted parallax distribution: S

Threshold value: th

The maximum value of parallax: Dmax

The minimum value of parallax: Dmin

Further, S-th,

Examples of the data are shown.

The amount of parallel movement (Shift) which is output can becalculated by the following expression.Shift=−Davg

Alternatively,Shift=−Dcenter

In step S204, the amount of parallel movement (Shift) of the weightedparallax distribution is calculated according to the above expression.

In step S205 subsequent thereto, the determining processing of thevirtual view point positions is performed.

The detailed sequence of this processing will be explained withreference to the flowchart of FIG. 24.

First, in step S221, correction of the weighted parallax distribution,i.e., parallel movement, is performed on the basis of the amount ofparallel movement obtained in step S204.

More specifically, as illustrated in FIG. 25,

(a) the weighted parallax distribution data are moved in parallel on thebasis of the amount of parallel movement obtained in step S203, and

(b) corrected weighted parallax distribution data are generated.

This parallax distribution correction processing can be expressed as thefollowing expression.

The parameters are defined as follows.

Shift: the amount of parallel movement (calculated in step S204)

(d): parallax distribution (weighted parallax distribution)

H(d)′: correction parallax distribution (corrected weighted parallaxdistribution)

At this occasion, the corrected weighted parallax distribution[H(d)′] iscalculated according to the following expression.H(d)′=H(d−Shift)

In step S222 subsequent thereto, the maximum value of parallax iscalculated by applying weighted parallax distribution informationcalculated in step S203. This processing is the same processing as theprocessing of steps S143 in the flow as illustrated in FIG. 14 explainedin the second embodiment.

The processing of step S223 subsequent thereto is the same processing asthe processing of step S121 in the flow of FIG. 5 explained in the firstembodiment.

In other words, calculation of the image qualities of the virtual viewpoint images is executed. This processing is the processing explainedwith reference to FIGS. 6 to 8 above.

In step S224 subsequent thereto, the virtual view point interval iscalculated by applying all of the following items:

the appropriate amount of parallax obtained in step S202,

the maximum value of parallax obtained in step S222, and

the image quality obtained in step S223.

This processing is the same processing as the processing of steps S165in the flow as illustrated in FIG. 19 explained in the third embodimentabove.

In the final step of S225, virtual view point positions are determinedby applying the virtual view point interval (G) calculated in step S224.

This determining processing of the virtual view point positions is thesame processing as the processing of step S123 of the flow of FIG. 5 ofthe first embodiment explained above. More specifically, this is thesame processing as the processing explained with reference to FIGS. 9 to13.

The determining processing of the virtual view point positions of stepS205 in the flow of FIG. 22 is performed in accordance with steps S221to S225 in the flow as illustrated in FIG. 24.

Subsequently, the generating processing of virtual view point images isexecuted in step S205 as illustrated in the flow of FIG. 24.

This processing is the same processing as the processing of steps S103in the flow as illustrated in FIG. 1 explained in the first embodimentabove.

In step S205, images corresponding to images observed from the virtualview point positions determined in step S204 are generated. In otherwords, the virtual view point images are generated. In step S204, forexample, a predetermined number of (for example, 10) virtual view pointsare determined, and in step S105, virtual view point imagescorresponding to the virtual view points are generated.

The virtual view point images are generated using the received standardLR images as explained with reference to FIG. 2 above. In other words,they are generated using the original left eye image (L image) and theoriginal right eye image (R image) for three-dimensional image display.

Subsequently, in step S207, the amount of parallel movement is correctedaccording to the virtual view point interval (G).

This processing will be explained with reference to FIG. 26.

The parameters are defined as follows.

Shift: the amount of parallel movement (value determined in step S204)

Shift′: the amount of parallel movement after correction

G: virtual view point interval (G determined in step S205 (step S224))

Under the setting of the above parameters, the corrected amount ofparallel movement Shift′ is calculated according to the followingexpression.−Shift′=Shift*G

FIG. 26 shows:

(a) parallax distribution between the received LR images,

(b) parallax distribution between adjacent virtual view points (solidline),

(c) parallax distribution to which the corrected amount of parallelmovement has been applied.

The parallax distribution between adjacent virtual view points asillustrated in (b) is changed depending on the virtual view pointinterval G. In other words, H_(E) (d)=H(d*G) holds, and the correctedparallax distribution HE (d)′ as illustrated in (c) can be expressedaccording to the following expression.H _(E)(d)′=H(d*G+Shift′)

Subsequently, in step S208, an image moved in parallel of a virtual viewpoint image is generated.

This processing will be explained with reference to FIG. 27.

The parameters are defined as follows.

Shift′: the amount of parallel movement after correction (valuedetermined in step S207)

L(i, x, y): the i-th virtual view point image before parallel movement

L(i, x, y)′: the i-th virtual view point image after parallel movement

Nfixshift: the number of virtual view point image of reference that isnot moved in parallel (user input)

Shift′ (i): the amount of parallel movement applied to the i-th virtualview point image

It should be noted that (x, y) in L(i, x, y), L(i, x, y)′ denotes apixel position constituting an image, and means a pixel value of eachimage.

On the basis of the above parameters, the amount of parallel movement[Shift′ (i)] applied to the i-th virtual view point image is calculatedby the following expression.Shift′(i)=Shift′*(i−Nfixshift))

Further, the i-th virtual view point image[L(i, x, y)′] after theparallel movement is calculated b the following expression.L(i,x,y)′=L(i,x+Shift′(i),y)

The above expression means that the virtual view point image moved inparallel can be generated by moving the pixel positions of the image inthe x direction (horizontal shift).

The example as illustrated in FIG. 27 is a figure showing an example ofexecution of parallel movement (shift) processing of each virtual viewpoint image in accordance with the following setting:

N=9,

Nfixshift=4, and

Shift′=1.5.

As described above, the virtual view point image generating unit of theimage processing apparatus according to the present embodimentdetermines virtual view point positions by means of processing in viewof at least one of the image qualities of the virtual view point images,the appropriate amount of parallax, and the image weights according toimage regions, calculates the amount of parallel movement on the basisof parallax distribution data calculated from the parallax information,executes moving processing of the parallax distribution between thevirtual view point images of the respective virtual view point positionson the basis of the amount of parallel movement calculated, andgenerates virtual view point images reflecting the moving processingresult of the parallax distribution data.

In the explanation about the present embodiment explained above, theexample of configuration has been explained in which the weightedparallax distribution is generated on the basis of the parallaxdistribution obtained from the received LR images, and the generatedweighted parallax distribution data are processed. However, thefollowing configuration may also be possible: processing may beperformed by applying parallax distribution data obtained from thereceived

LR images as they are, without generating the weighted parallaxdistribution.

6. Example of Configuration of Image Processing Apparatus

Subsequently, an example of configuration of an image processingapparatus executing the processing according to the embodimentsexplained above will be explained with reference to FIG. 28.

An image processing apparatus 300 as illustrated in FIG. 28 includes aleft eye image (L image) input unit 301, a right eye image (R image)input unit 302, a parallax information generating unit 303, a weightinformation generating unit 304, a virtual view point image generatingunit 305, a display control unit 306, and images generated by the imageprocessing apparatus 300 is output to a display unit 310.

In the configuration as illustrated in FIG. 28, the display unit 310 isshown as an external configuration of the image processing apparatus300. Alternatively, a configuration is also possible in which thedisplay unit 310 is provided within the image processing apparatus 300.

The configuration as illustrated in FIG. 28 shows primary elements ofthe image processing apparatus 300, and the image processing apparatus300 includes not only the elements as illustrated in the figure but alsoa control unit having a program execution function such as a CPUexecuting data processing control, a program executed by the controlunit, a storage unit storing various kinds of parameters, and an inputunit for inputting parameters, image data, and the like.

The left eye image (L image) input unit 301 and the right eye image (Rimage) input unit 302 inputs a left eye image (L image) and right eyeimage (R image) for three-dimensional (3D) image display generated inadvance.

The parallax information generating unit 303 receives the left eye image(L image) and the right eye image (R image), and generates parallaxinformation on the basis of these images.

As described above, the parallax information corresponds to adisplacement between images of the same subject included in standard LRimages (pixel displacement in a horizontal direction), and isinformation corresponding to a distance of a subject. More specifically,for example, data having parallax information (subject distanceinformation) in units of pixels are generated.

As described above, the acquisition of the parallax information isexecuted according to, for example, an existing method as follows.

(a) block matching-based parallax information acquisition processing

(b) DP (dynamic programming) matching-based parallax informationacquisition processing

(c) segmentation-based parallax information acquisition processing

(d) learning-based parallax information acquisition processing

(e) Parallax information acquisition processing of a combination of theabove methods

For example, the parallax information is obtained according to any oneof the above methods (a) to (e).

The weight information generating unit 304 uses any one of the L imageand the R image to generate weight information representing imageweights in units of image regions. For example, the weight informationis information in which larger weights are set in image regions that arelikely to attract attention of an observer, which are, morespecifically, a central portion of an image and a facial image region ofa person.

More specifically, as explained with reference to FIGS. 15 to 17 above,

(1) weight information where weights are set according to image regions(a larger weight is set in a central region)

(2) weight information weights are set according to subjects (a largerweight is set in a person region)

(3) weight information in which both of processing of (1) and (2)explained above are combined

For example, it is such weight information.

The virtual view point image generating unit 305 generates virtual viewpoint images by receiving the following information:

the L image from the left eye image (L image) input unit 301,

the R image from the right eye image (R image) input unit 302,

the parallax information from the parallax information generating unit303, and

the weight information from the weight information generating unit 304.

The virtual view point image generating unit 305 executes thedetermining processing of a virtual view point interval and thedetermining processing of the virtual view point positions in accordancewith any one of the methods of the first to the fifth embodimentsexplained above, and generates virtual view point images correspondingto the determined virtual view point positions.

More specifically, for example, generating processing of the virtualview point images is executed according to processing of any one of thefollowing items.

(1) generating processing of the virtual view point images correspondingto the virtual view point positions determined in view of the imagequality (first embodiment)

(2) generating processing of the virtual view point images correspondingto the virtual view point positions determined on the basis of theappropriate amount of parallax and the image weights (second embodiment)

(3) generating processing of the virtual view point images correspondingto the virtual view point positions determined on the basis of the imagequality, the appropriate amount of parallax, and the image weights(third embodiment)

(4) generating processing of the virtual view point images correspondingto the virtual view point positions of non-regular interval (fourthembodiment)

(5) generating processing of the virtual view point image using theshift processing (fifth embodiment)

The virtual view point image generating unit 305 generates multi-viewpoint images of, e.g., N different view points, on the basis of theabove processing, and outputs the multi-view point images to the displaycontrol unit 306.

The display control unit 306 generates display information according tothe display unit 310 on the basis of the multi-view point imagesgenerated by the virtual view point image generating unit 305, andoutputs the display information to the display unit 310.

As described above, the display image generated by the image processingapparatus according to an embodiment of the present disclosure is adisplay image of a naked eye 3D display apparatus, with which a user canview a stereoscopic image without wearing glasses.

The display unit 310 is a display unit for naked eye 3D display. Thedisplay unit 310 includes, for example, a lenticular sheet or a parallaxbarrier (parallax barrier) on a display surface, which can controlimages entering into the left eye and the right eye in accordance with aviewing/listening position.

For example, as explained with reference to FIG. 3 above, the displaycontrol unit 306 generates an image obtained by interleaving the imagesof the N view points generated by the virtual view point imagegenerating unit 305, and outputs the image to the display unit 310.

It should be noted that the display control unit 306 generates displayinformation in accordance with the display configuration of the displayunit 310.

It should be noted that the image processing apparatus may also beconfigured as, for example, an image-capturing apparatus such as acamera having an image-capturing unit and a display apparatus such as aPC and a television set, and when it is configured as such apparatus,the image processing apparatus has a configuration having functionsaccording to each apparatus.

For example, in a case of a camera, the image processing apparatus isconfigured to include an image-capturing unit for taking LR images asimages from different view points and generate multi-view point imagesusing the LR images received from the image-capturing unit.

7. Summary of Configuration of the Present Disclosure

It should be understood by those skilled in the art that variousmodifications, combinations, sub-combinations and alterations may occurdepending on design requirements and other factors insofar as they arewithin the scope of the appended claims or the equivalents thereof.

Additionally, the present technology may also be configured as below.

-   (1) An image processing apparatus including:

a left eye image input unit configured to input a left eye image (Limage) which is a left eye image signal applied to three-dimensionalimage display;

a right eye image input unit configured to input a right eye image (Rimage) which is a right eye image signal applied to three-dimensionalimage display;

a parallax information generating unit configured to generate parallaxinformation from the left eye image (L image) and the right eye image (Rimage); and

a virtual view point image generating unit configured to receive theleft eye image (L image), the right eye image (R image), and theparallax information, and generate virtual view point images including aview point image other than view points of the received LR images,

wherein the virtual view point image generating unit determines virtualview point positions by means of processing in view of at least one ofimage qualities of virtual view point images, an appropriate amount ofparallax, or an image weight according to an image region, and generatesthe virtual view point images corresponding to the determined virtualview point positions.

-   (2) The image processing apparatus according to (1),

wherein the virtual view point image generating unit calculates an imagequality evaluation value Q indicating an image quality of a virtual viewpoint image, calculates a virtual view point interval G by applying thecalculated image quality evaluation value Q, and determines the virtualview point position on the basis of the calculated virtual view pointinterval G.

-   (3) The image processing apparatus according to (2),

wherein the virtual view point image generating unit calculates theimage quality evaluation value Q by applying information of at least oneof reliability degree information of the parallax information or thegenerated virtual view point image information.

-   (4) The image processing apparatus according to any one of (1) to    (3),

wherein the virtual view point image generating unit calculates, as anappropriate amount of parallax, a smaller value of a fusional parallaxamount and a crosstalk allowable amount, calculates a virtual view pointinterval G by applying the calculated appropriate amount of parallax,and determines the virtual view point position on the basis of thecalculated virtual view point interval G.

-   (5) The image processing apparatus according to any one of (1) to    (4),

wherein the virtual view point image generating unit calculates, as anappropriate amount of parallax, a smaller value of a fusional parallaxamount and a crosstalk allowable amount, calculates a virtual view pointinterval G by applying the calculated appropriate amount of parallax,and determines the virtual view point position on the basis of thecalculated virtual view point interval G.

-   (6) The image processing apparatus according to any one of (1) to    (5),

wherein the image processing apparatus includes a weight informationgenerating unit configured to calculate image weight informationaccording to an image region, and

wherein the virtual view point image generating unit calculates aweighted parallax distribution obtained by correcting the parallaxinformation by applying the image weight information, calculates avirtual view point interval G by applying the appropriate amount ofparallax and a maximum value of parallax calculated from the calculatedweighted parallax distribution, and determines the virtual view pointposition on the basis of the calculated virtual view point interval G.

-   (7) The image processing apparatus according to any one of (1) to    (6),

wherein the weight information generating unit generates image weightinformation in which a weight in unit of image region is set accordingto a position of an image or image weight information according to asubject included in an image.

-   (8) The image processing apparatus according to any one of (1) to    (7),

wherein the virtual view point image generating unit determines a firstvirtual view point position by means of processing in view of at leastone of an image quality of a virtual view point image, an appropriateamount of parallax, or an image weight according to an image region,determines a second virtual view point position of non-regular intervalby means of non-linear mapping processing performed on the determinedfirst virtual view point position, and generates a virtual view pointimage corresponding to the determined second virtual view point positionof the non-regular interval.

-   (9) The image processing apparatus according to any one of (1) to    (8),

wherein the virtual view point image generating unit determines thevirtual view point position by means of processing in view of at leastone of an image quality of the virtual view point image, an appropriateamount of parallax, or an image weight according to an image region,calculates an amount of parallel movement on the basis of parallaxdistribution data calculated from the parallax information, executesmoving processing of the parallax distribution between the virtual viewpoint images of the respective virtual view point positions on the basisof the calculated amount of parallel movement, and generates virtualview point images reflecting a moving processing result of the parallaxdistribution data.

Further, a method of processing executed by the above apparatus and thelike and a program executing the processing are also included in theconfiguration of the present disclosure.

The series of processing explained in the specification can be executedby either hardware, software or a composite configuration of them both.When the processing is executed by software, a program having theprocessing sequence recorded therein can be installed and executed in amemory within a computer incorporated into dedicated hardware, or theprogram can be installed and executed in a general-purpose computercapable of executing various kinds of processing. For example, theprogram can be recorded to a recording medium in advance. The programcan be installed to the computer from a recording medium. Alternatively,the program can be received via a network such as a LAN (Local AreaNetwork) and the Internet, and the program can be installed to arecording medium such as an internal hard disk.

Various kinds of processing described in the specification are notlimited to execution in time series as described therein. Alternatively,various kinds of processing can be executed in parallel or individually,in accordance with the performance of processing of the apparatusexecuting the processing or as necessary. In this specification, asystem is a logical configuration of a set of multiple apparatuses, andan apparatus of each configuration is not necessarily limited to beprovided within the same housing.

As described above, according to a configuration of an embodiment of thepresent disclosure, a configuration for generating multi-view pointimages based on LR images of three-dimensional images is achieved.

More specifically, for example, a virtual view point image generatingunit is provided, wherein the virtual view point image generating unitreceives a left eye image (L image) and a right eye image (R image)which are applied to three-dimensional image display, generates parallaxinformation on the basis of the left eye image (L image) and the righteye image (R image), and uses the LR image and the parallax informationto generate virtual view point images including view point images otherthan the view points of the received LR images. The virtual view pointimage generating unit determines the virtual view point positions bymeans of processing in view of at least one of an image quality of avirtual view point image, an appropriate amount of parallax determinedin view of a fusional parallax amount and a crosstalk allowable amount,and an image weight according to an image region of a subject and thelike and a position of an image, and generates virtual view point imagescorresponding to the determined virtual view point positions.

With such processing, optimum virtual view point images according torespective observation positions, i.e., high-quality virtual view pointimages of comfortable parallax ranges, can be generated.

The present disclosure contains subject matter related to that disclosedin Japanese Priority Patent Application JP 2011-173268 filed in theJapan Patent Office on Aug. 8, 2011, the entire content of which ishereby incorporated by reference.

What is claimed is:
 1. An image processing apparatus comprising: a lefteye image input unit configured to input a left eye image (L image)which is a left eye image signal applied to three-dimensional imagedisplay; a right eye image input unit configured to input a right eyeimage (R image) which is a right eye image signal applied tothree-dimensional image display; a parallax information generating unitconfigured to generate parallax information from the left eye image (L,image) and the right eye image (R image); and a virtual view point imagegenerating unit configured to receive the left eye image (L image), theright eye image (R image), and the parallax information, and generatevirtual view point images including a view point image other than viewpoints of the received LR images, wherein the virtual view point imagegenerating unit determines virtual view point positions by means ofprocessing in view of at least one of image qualities of virtual viewpoint images, an appropriate amount of parallax, or an image weightaccording to an image region, and generates the virtual view pointimages corresponding to the determined virtual view point positions,wherein the virtual view point image generating unit determines a firstvirtual view point position by means of processing in view of at leastone of an image quality of a virtual view point image, an appropriateamount of parallax, or an image weight according to an image region,determines a second virtual view point position of non-regular intervalby means of non-linear mapping processing performed on the determinedfirst virtual view point position, and generates a virtual view pointimage corresponding to the determined second virtual view point positionof the non-regular interval, and wherein the left eye image input unit,the right eye image input unit, the parallax information generatingunit, and the virtual view point image generating unit are eachimplemented via at least one processor.
 2. The image processingapparatus according to claim 1, wherein the virtual view point imagegenerating unit calculates an image quality evaluation value Qindicating an image quality of a virtual view point image, calculates avirtual view point interval G by applying the calculated image qualityevaluation value Q, and determines the virtual view point position onthe basis of the calculated virtual view point interval G.
 3. The imageprocessing apparatus according to claim 2, wherein the virtual viewpoint image generating unit calculates the image quality evaluationvalue Q by applying information of at least one of reliability degreeinformation of the parallax information or the generated virtual viewpoint image information.
 4. The image processing apparatus according toclaim 1, wherein the virtual view point image generating unitcalculates, as an appropriate amount of parallax, a smaller value of afusional parallax amount and a crosstalk allowable amount, calculates avirtual view point interval G by applying the calculated appropriateamount of parallax, and determines the virtual view point position onthe basis of the calculated virtual view point interval G.
 5. The imageprocessing apparatus according to claim 1, wherein the image processingapparatus includes a weight information generating unit configured tocalculate image weight information according to an image region, whereinthe virtual view point image generating unit calculates a weightedparallax distribution obtained by correcting the parallax information byapplying the image weight information, calculates a virtual view pointinterval G by applying the appropriate amount of parallax and a maximumvalue of parallax calculated from the calculated weighted parallaxdistribution, and determines the virtual view point position on thebasis of the calculated virtual view point interval G, and wherein theweight information generating unit is implemented via at least oneprocessor.
 6. The image processing apparatus according to claim 5,wherein the weight information generating unit generates image weightinformation in which a weight in unit of image region is set accordingto a position of an image or image weight information according to asubject included in an image.
 7. The image processing apparatusaccording to claim 1, wherein the virtual view point image generatingunit determines the virtual view point position by means of processingin view of at least one of an image quality of the virtual view pointimage, an appropriate amount of parallax, or an image weight accordingto an image region, calculates an amount of parallel movement on thebasis of parallax distribution data calculated from the parallaxinformation, executes moving processing of the parallax distributionbetween the virtual view point images of the respective virtual viewpoint positions on the basis of the calculated amount of parallelmovement, and generates virtual view point images reflecting a movingprocessing result of the parallax distribution data.
 8. Animage-capturing apparatus comprising: an image-capturing unit configuredto capture a left eye image (L image) which is a left eye image signaland a right eye image (R image) which is a right eye image signal, whichare applied to three-dimensional image display; a left eye image inputunit configured to input, from the image-capturing unit, the left eyeimage (L image) which is the left eye image signal applied to thethree-dimensional image display; a right eye image input unit configuredto input, from the image-capturing unit, the right eye image (R image)which is the right eye image signal applied to the three-dimensionalimage display; a parallax information generating unit configured togenerate parallax information from the left eye image (L image) and theright eye image (R image); and a virtual view point image generatingunit configured to receive the left eye image (L image), the right eyeimage (R image), and the parallax information, and generate virtual viewpoint images including a view point image other than view points of thereceived LR images, wherein the virtual view point image generating unitdetermines virtual view point positions by means of processing in viewof at least one of image qualities of virtual view point images, anappropriate amount of parallax, or an image weight according to an imageregion, and generates the virtual view point images corresponding to thedetermined virtual view point positions, wherein the virtual view pointimage generating unit determines a first virtual view point position bymeans of processing in view of at least one of an image quality of avirtual view point image, an appropriate amount of parallax, or an imageweight according to an image region, determines a second virtual viewpoint position of non-regular interval by means of non-linear mappingprocessing performed on the determined first virtual view pointposition, and generates a virtual view point image corresponding to thedetermined second virtual view point position of the non-regularinterval, and wherein the image capturing unit, the left eye image inputunit, the right eye image input unit, the parallax informationgenerating unit, and the virtual view point image generating unit areeach implemented via at least one processor.
 9. An image processingmethod with which an image processing apparatus generates multi-viewpoint images, the image processing method comprising: inputting, by aleft eye image input unit, a left eye image (L image) which is a lefteye image signal applied to three-dimensional image display; inputting,by a right eye image input unit, a right eye image (R image) which is aright eye image signal applied to three-dimensional image display;generating, by a parallax information generating unit, parallaxinformation from the left eye image (L image) and the right eye image (Rimage); and receiving, by a virtual view point image generating unit,the left eye image (L image), the right eye image (R image), and theparallax information, and generating virtual view point images includinga view point image other than view points of the received LR images,wherein, in the virtual view point image generating step, virtual viewpoint positions are determined by means of processing in view of atleast one of image qualities of virtual view point images, anappropriate amount of parallax, or an image weight according to an imageregion, and the virtual view point images corresponding to thedetermined virtual view point positions are generated, wherein thevirtual view point image generating unit determines a first virtual viewpoint position by means of processing in view of at least one of animage quality of a virtual view point image, an appropriate amount ofparallax, or an image weight according to an image region, determines asecond virtual view point position of non-regular interval by means ofnon-linear mapping processing performed on the determined first virtualview point position, and generates a virtual view point imagecorresponding to the determined second virtual view point position ofthe non-regular interval, and wherein the left eye image input unit, theright eye image input unit, the parallax information generating unit,and the virtual view point image generating unit are each implementedvia at least one processor.
 10. A non-transitory computer-readablemedium having embodied thereon a program, which when executed by atleast one processor of an image processing apparatus causes the imageprocessing apparatus to execute a method of generating multi-view pointimages, the method comprising: inputting a left eye image (L image)which is a left eye image signal applied to three-dimensional imagedisplay; inputting a right eye image (R image) which is a right eyeimage signal applied to three-dimensional image display; generatingparallax information from the left eye image (L image) and the right eyeimage (R image); and receiving the left eye image (L image), the righteye image (R image), and the parallax information, and generatingvirtual view point images including a view point image other than viewpoints of the received LR images, wherein, in generating the virtualview point images, virtual view point positions are determined by meansof processing in view of at least one of image qualities of virtual viewpoint images, an appropriate amount of parallax, or an image weightaccording to an image region, and the virtual view point imagescorresponding to the determined virtual view point positions aregenerated, and wherein a first virtual view point position is determinedby means of processing in view of at least one of an image quality of avirtual view point image, an appropriate amount of parallax, or an imageweight according to an image region, a second virtual view pointposition of non-regular interval is determined by means of non-linearmapping processing performed on the determined first virtual view pointposition, and a virtual view point image corresponding to the determinedsecond virtual view point position of the non-regular interval isgenerated.